2024-03-12T03:00:18.526 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T03:00:18.526 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862 --verbose -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862/orig.config.yaml 2024-03-12T03:00:18.581 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T03:00:18.678 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862 branch: main description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591862' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/reef ' name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVF0YvyzhzCRwLRGv3CVYU883ajunQVhb3K+nZMFHHWog9mEDJNR9E5d5OndRoXiAWhGhwKsKgPsbrq18T5bR4= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGdeBfwI5tI5+DIeBLgsIfbfseyNiNrXbFqN85+hvWLqJoRwTeCkG2KHpZat8/o6wlt81KlndtaoaTWXHx7K/QA= tasks: - install: branch: reef exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: compiled_cephadm_branch: reef conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:reef roleless: true - print: '**** done end installing reef cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-12T03:00:18.678 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-12T03:00:18.679 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-12T03:00:18.679 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T03:00:18.681 INFO:teuthology.task.internal:Checking packages... 2024-03-12T03:00:18.703 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-12T03:00:18.703 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:00:18.703 INFO:teuthology.packaging:ref: None 2024-03-12T03:00:18.703 INFO:teuthology.packaging:tag: None 2024-03-12T03:00:18.704 INFO:teuthology.packaging:branch: main 2024-03-12T03:00:18.704 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:00:18.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-12T03:00:18.924 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-12T03:00:18.926 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T03:00:18.932 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T03:00:18.932 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T03:00:18.939 INFO:teuthology.task.internal:Saving configuration 2024-03-12T03:00:18.964 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T03:00:18.970 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T03:00:18.997 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:54:37.197875', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVF0YvyzhzCRwLRGv3CVYU883ajunQVhb3K+nZMFHHWog9mEDJNR9E5d5OndRoXiAWhGhwKsKgPsbrq18T5bR4='} 2024-03-12T03:00:19.018 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:54:37.196400', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGdeBfwI5tI5+DIeBLgsIfbfseyNiNrXbFqN85+hvWLqJoRwTeCkG2KHpZat8/o6wlt81KlndtaoaTWXHx7K/QA='} 2024-03-12T03:00:19.018 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T03:00:19.025 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-12T03:00:19.025 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-12T03:00:19.025 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T03:00:19.096 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6a20e9f0d0>, signals=[15]) 2024-03-12T03:00:19.096 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T03:00:19.103 INFO:teuthology.task.internal:Opening connections... 2024-03-12T03:00:19.103 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:00:19.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:00:19.208 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-03-12T03:00:19.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:00:19.299 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T03:00:19.307 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-03-12T03:00:19.325 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-03-12T03:00:19.325 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-03-12T03:00:19.382 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-03-12T03:00:19.382 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-03-12T03:00:19.382 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-03-12T03:00:19.382 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:00:19.382 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-03-12T03:00:19.382 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:00:19.383 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:00:19.384 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-03-12T03:00:19.405 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-03-12T03:00:19.423 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-03-12T03:00:19.423 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-12T03:00:19.480 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-12T03:00:19.480 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:00:19.481 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:00:19.481 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-12T03:00:19.530 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T03:00:19.540 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T03:00:19.545 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T03:00:19.545 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:00:19.548 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:00:19.564 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T03:00:19.570 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T03:00:19.571 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:00:19.604 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:00:19.683 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T03:00:19.737 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T03:00:19.752 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-12T03:00:19.752 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-12T03:00:19.752 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:00:19.753 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:00:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:00:19.768 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-12T03:00:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:00:19.769 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-12T03:00:19.825 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:00:19.825 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.14.0-361.el9.x86_64 2024-03-12T03:00:19.826 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2024-03-12T03:00:19.827 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:00:19.827 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-361.el9.x86_64 2024-03-12T03:00:19.827 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-03-12T03:00:21.953 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-12T03:00:22.908 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-12T03:00:23.569 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-12T03:00:24.300 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-12T03:00:26.150 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-03-12T03:00:28.325 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-12T03:00:32.467 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-12T03:00:34.018 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-03-12T03:00:37.061 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-03-12T03:00:38.058 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 581 kB/s | 8.0 MB 00:14 2024-03-12T03:00:41.275 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 19 MB 00:02 2024-03-12T03:00:42.194 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 5.0 kB/s | 1.7 kB 00:00 2024-03-12T03:00:44.400 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:00:44.400 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:00:44.497 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:00:44.508 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:00:44.508 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-12T03:00:44.508 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:00:44.508 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2024-03-12T03:00:44.509 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:00:44.510 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 100 M 2024-03-12T03:00:44.510 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 123 M 2024-03-12T03:00:44.510 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:00:45.070 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 12 MB/s | 6.0 MB 00:00 2024-03-12T03:00:45.354 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 27 MB/s | 21 MB 00:00 2024-03-12T03:00:46.288 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 23 MB/s | 39 MB 00:01 2024-03-12T03:00:46.676 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-03-12T03:00:46.878 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 18 MB/s | 33 MB 00:01 2024-03-12T03:00:46.881 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:00:46.881 INFO:teuthology.orchestra.run.smithi118.stdout:Total 42 MB/s | 100 MB 00:02 2024-03-12T03:00:47.764 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:00:48.143 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:00:48.143 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:00:48.476 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-03-12T03:00:49.507 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:00:49.507 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:00:50.230 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-12T03:00:52.448 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:00:53.103 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:00:53.239 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:00:55.203 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-12T03:00:56.032 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:00:56.215 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:00:57.409 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:00:57.410 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:00:57.498 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:00:57.509 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:00:57.510 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:00:57.511 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:00:57.511 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-03-12T03:00:57.511 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:00:57.511 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 100 M 2024-03-12T03:00:57.511 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 123 M 2024-03-12T03:00:57.512 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:01:00.952 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:01:01.074 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:01.429 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 1.6 MB/s | 6.0 MB 00:03 2024-03-12T03:01:05.849 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:10.761 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 1.6 MB/s | 21 MB 00:12 2024-03-12T03:01:16.602 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 2.2 MB/s | 33 MB 00:15 2024-03-12T03:01:19.519 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 1.8 MB/s | 39 MB 00:21 2024-03-12T03:01:19.519 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:01:19.519 INFO:teuthology.orchestra.run.smithi177.stdout:Total 4.5 MB/s | 100 MB 00:22 2024-03-12T03:01:20.402 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:01:20.772 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:01:20.773 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:01:22.111 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:01:22.111 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:01:25.093 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:01:25.746 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:01:25.878 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:01:28.666 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:01:28.872 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:01:33.601 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:01:33.733 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:35.725 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:35.725 INFO:teuthology.orchestra.run.smithi118.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:01:35.725 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:01:35.812 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:36.271 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:36.271 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:01:36.271 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:01:36.271 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:01:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:36.705 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:01:36.706 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:01:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:01:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:01:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:01:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:01:36.706 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:01:36.706 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:01:37.208 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:01:37.345 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:01:37.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:01:37.346 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:01:37.346 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:01:37.346 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:01:37.346 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-12T03:01:37.346 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:01:37.346 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:01:37.466 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:01:37.466 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:01:37.466 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-03-12T03:01:37.505 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-03-12T03:01:37.505 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-03-12T03:01:37.505 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2024-03-12T03:01:37.505 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 68.el9 2024-03-12T03:01:37.505 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-03-12T03:01:37.505 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8264254 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:01:37.506 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:01:37.507 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:01:37.507 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-03-12T03:01:37.507 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:01:37.507 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:01:37.507 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:01:37.508 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:01:37.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:01:38.121 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-03-12T03:01:38.447 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:01:39.508 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:01:39.541 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-03-12T03:01:39.543 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-03-12T03:01:39.558 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.EJ0WBNkfaw 2024-03-12T03:01:39.558 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EJ0WBNkfaw 2024-03-12T03:01:39.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.EJ0WBNkfaw 2024-03-12T03:01:39.763 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.EJ0WBNkfaw is 6KB 2024-03-12T03:01:39.812 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.EJ0WBNkfaw 2024-03-12T03:01:39.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:01:39.892 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:01:39.893 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:01:39.893 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:01:39.893 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:01:39.894 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:01:40.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-03-12T03:02:07.564 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:02:07.565 INFO:teuthology.orchestra.run.smithi177.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:02:07.565 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:02:07.648 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:02:08.035 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:02:08.036 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:02:08.036 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:02:08.036 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:02:08.482 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:02:08.483 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:02:08.965 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:02:09.106 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:02:09.106 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:02:09.106 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:02:09.107 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:02:09.107 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:02:09.107 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-12T03:02:09.107 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:02:09.107 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:02:09.229 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:02:09.229 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:02:09.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-03-12T03:02:09.271 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:02:09.272 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:02:09.273 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:02:09.274 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:02:09.274 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:02:09.274 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:02:09.895 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-03-12T03:02:10.035 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-12T03:02:10.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-12T03:02:10.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:02:11.325 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:02:11.357 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-03-12T03:02:11.368 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-03-12T03:02:11.391 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.Ac8h1kbj2g 2024-03-12T03:02:11.391 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ac8h1kbj2g 2024-03-12T03:02:11.463 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.Ac8h1kbj2g 2024-03-12T03:02:11.604 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.Ac8h1kbj2g is 6KB 2024-03-12T03:02:11.652 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.Ac8h1kbj2g 2024-03-12T03:02:11.667 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:02:11.731 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:02:11.731 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:02:11.731 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:02:11.731 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:02:11.732 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:02:11.887 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-03-12T03:02:28.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-12T03:02:37.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-12T03:02:37.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:02:40.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-12T03:02:41.890 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-12T03:02:41.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T03:02:41.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:02:52.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-12T03:02:52.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:00.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T03:03:09.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T03:03:09.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:12.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T03:03:24.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T03:03:24.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:03:52.489 DEBUG:teuthology.orchestra.remote:timed out 2024-03-12T03:03:56.188 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-12T03:03:56.552 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T03:03:56.552 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:03:56.552 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-12T03:03:56.566 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:03:56.567 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:03:56.567 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:03:56.567 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:03:57.567 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-12T03:03:57.568 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:03:57.640 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-03-12T03:03:57.679 DEBUG:teuthology.parallel:result is None 2024-03-12T03:04:07.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-12T03:04:07.492 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:04:07.919 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-12T03:04:08.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-12T03:04:08.316 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:04:08.316 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-12T03:04:08.332 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:04:08.332 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:04:08.332 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:04:08.332 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:04:09.333 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-12T03:04:09.333 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:04:09.408 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-03-12T03:04:09.438 DEBUG:teuthology.parallel:result is None 2024-03-12T03:04:09.439 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T03:04:09.446 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T03:04:09.446 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:04:09.449 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:04:09.467 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T03:04:09.474 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T03:04:09.484 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T03:04:09.484 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:04:09.506 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:04:09.614 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T03:04:09.621 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T03:04:09.621 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:04:09.624 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:04:09.662 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:04:09.663 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:04:09.676 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:04:09.676 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:04:09.678 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T03:04:09.690 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T03:04:09.691 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:04:09.721 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:04:09.752 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T03:04:09.762 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T03:04:09.762 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:04:09.791 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:04:09.808 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:04:09.889 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:04:09.936 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:04:10.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:04:10.060 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:04:10.060 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:04:10.128 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:04:10.173 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:04:10.233 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:04:10.276 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:04:10.321 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:04:10.321 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:04:10.391 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-03-12T03:04:10.393 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-03-12T03:04:10.432 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:04:10.476 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:04:10.844 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T03:04:10.853 INFO:teuthology.task.internal:Starting timer... 2024-03-12T03:04:10.853 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T03:04:10.863 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T03:04:10.871 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-12T03:04:10.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-03-12T03:04:10.945 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-03-12T03:04:10.947 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-03-12T03:04:10.998 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-03-12T03:04:11.000 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T03:04:11.000 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-03-12T03:04:11.017 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-03-12T03:04:11.018 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-03-12T03:04:11.044 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-03-12T03:04:11.044 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi118.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-03-12T03:04:11.044 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-12T03:04:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:04:11.085 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-12T03:04:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:04:11.113 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T03:04:11.113 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:04:11.161 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:04:11.204 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T03:04:11.214 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-12T03:04:11.219 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-12T03:04:11.340 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T03:04:11.354 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T03:04:11.355 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi118.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T03:08:27.535 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi118.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-03-12T03:08:27.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-12T03:08:27.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:08:27.627 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-12T03:08:27.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-12T03:08:27.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T03:08:27.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:08:27.797 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-12T03:08:27.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T03:08:27.877 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T03:08:27.888 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T03:08:27.888 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:08:27.889 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:08:27.891 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:08:27.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:08:27.919 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:08:27.935 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:08:27.946 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:08:27.962 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:08:27.964 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-03-12T03:08:27.977 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-03-12T03:08:27.992 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-03-12T03:08:27.994 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:08:28.006 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-03-12T03:08:28.010 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:08:28.022 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:08:28.038 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:08:28.066 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:08:28.070 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:08:28.070 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-12T03:08:28.070 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.070 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.070 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.070 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.090 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:08:28.093 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:08:28.093 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-12T03:08:28.093 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.093 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.094 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.094 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:08:28.094 INFO:teuthology.run_tasks:Running task install... 2024-03-12T03:08:28.102 DEBUG:teuthology.task.install:project ceph 2024-03-12T03:08:28.102 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}} 2024-03-12T03:08:28.102 DEBUG:teuthology.task.install:config {'branch': 'reef', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-12T03:08:28.102 INFO:teuthology.task.install:Using flavor: default 2024-03-12T03:08:28.107 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-12T03:08:28.107 INFO:teuthology.task.install:extra packages: [] 2024-03-12T03:08:28.107 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-12T03:08:28.108 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:08:28.108 INFO:teuthology.packaging:ref: None 2024-03-12T03:08:28.108 INFO:teuthology.packaging:tag: None 2024-03-12T03:08:28.108 INFO:teuthology.packaging:branch: reef 2024-03-12T03:08:28.108 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:08:28.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-12T03:08:28.110 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'reef', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-12T03:08:28.110 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:08:28.110 INFO:teuthology.packaging:ref: None 2024-03-12T03:08:28.110 INFO:teuthology.packaging:tag: None 2024-03-12T03:08:28.110 INFO:teuthology.packaging:branch: reef 2024-03-12T03:08:28.111 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:08:28.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-12T03:08:28.363 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/ 2024-03-12T03:08:28.363 INFO:teuthology.task.install.rpm:Package version is 18.2.2-751.g62a11b3c 2024-03-12T03:08:28.425 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/ 2024-03-12T03:08:28.425 INFO:teuthology.task.install.rpm:Package version is 18.2.2-751.g62a11b3c 2024-03-12T03:08:28.479 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T03:08:28.479 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:08:28.479 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T03:08:28.508 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-12T03:08:28.508 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:08:28.508 INFO:teuthology.packaging:ref: None 2024-03-12T03:08:28.508 INFO:teuthology.packaging:tag: None 2024-03-12T03:08:28.508 INFO:teuthology.packaging:branch: reef 2024-03-12T03:08:28.508 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:08:28.509 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T03:08:28.576 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T03:08:28.595 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-12T03:08:28.595 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:08:28.595 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-12T03:08:28.624 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-12T03:08:28.624 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:08:28.624 INFO:teuthology.packaging:ref: None 2024-03-12T03:08:28.624 INFO:teuthology.packaging:tag: None 2024-03-12T03:08:28.624 INFO:teuthology.packaging:branch: reef 2024-03-12T03:08:28.624 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:08:28.624 DEBUG:teuthology.orchestra.run.smithi177:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/reef/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-12T03:08:28.655 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T03:08:28.692 DEBUG:teuthology.orchestra.run.smithi177:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-12T03:08:28.721 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-03-12T03:08:28.722 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-03-12T03:08:28.772 DEBUG:teuthology.orchestra.run.smithi177:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-12T03:08:28.838 INFO:teuthology.orchestra.run.smithi177.stdout:check_obsoletes = 1 2024-03-12T03:08:28.840 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum clean all 2024-03-12T03:08:28.958 INFO:teuthology.orchestra.run.smithi118.stdout:74 files removed 2024-03-12T03:08:28.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2024-03-12T03:08:29.078 INFO:teuthology.orchestra.run.smithi177.stdout:74 files removed 2024-03-12T03:08:29.106 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-radosgw 2024-03-12T03:08:29.902 INFO:teuthology.orchestra.run.smithi177.stdout:ceph packages for x86_64 163 kB/s | 77 kB 00:00 2024-03-12T03:08:29.954 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 117 kB/s | 77 kB 00:00 2024-03-12T03:08:30.471 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 21 kB/s | 9.9 kB 00:00 2024-03-12T03:08:30.911 INFO:teuthology.orchestra.run.smithi177.stdout:ceph noarch packages 10 kB/s | 9.9 kB 00:00 2024-03-12T03:08:31.455 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 2.0 kB/s | 1.9 kB 00:00 2024-03-12T03:08:31.912 INFO:teuthology.orchestra.run.smithi177.stdout:ceph source packages 1.9 kB/s | 1.9 kB 00:00 2024-03-12T03:08:31.982 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-12T03:08:32.447 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-12T03:08:33.457 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-03-12T03:08:33.989 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-12T03:08:35.292 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.0 MB 00:01 2024-03-12T03:08:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-12T03:08:37.063 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-03-12T03:08:53.474 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-12T03:09:02.215 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 289 kB/s | 6.1 MB 00:21 2024-03-12T03:09:02.470 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 1.1 MB/s | 6.1 MB 00:05 2024-03-12T03:09:04.974 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-12T03:09:05.154 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-03-12T03:09:07.156 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-12T03:09:07.911 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 21 MB 00:02 2024-03-12T03:09:12.172 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-12T03:09:12.875 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-12T03:09:14.734 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:09:14.745 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:09:14.745 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:09:14.745 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:09:14.745 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:09:14.745 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 7.7 M 2024-03-12T03:09:14.745 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrading: 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: librados2 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 3.3 M 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 3.0 M 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 5.2 M 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 18 M 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 25 k 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 659 k 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T03:09:14.746 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 474 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 4.5 M 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 45 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 126 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 161 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 321 k 2024-03-12T03:09:14.747 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 299 k 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 99 k 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout:Install 28 Packages 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout:Upgrade 2 Packages 2024-03-12T03:09:14.748 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:14.749 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 53 M 2024-03-12T03:09:14.749 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:09:16.045 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 7.7 M 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 3.3 M 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 3.0 M 2024-03-12T03:09:16.052 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 5.2 M 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 18 M 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 25 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 659 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-12T03:09:16.053 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 474 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 4.5 M 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 45 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 126 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 161 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 321 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 299 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 99 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-12T03:09:16.054 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout:Install 28 Packages 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 53 M 2024-03-12T03:09:16.055 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:09:16.448 INFO:teuthology.orchestra.run.smithi177.stdout:(1/30): ceph-radosgw-18.2.2-751.g62a11b3c.el9.x 4.7 MB/s | 7.7 MB 00:01 2024-03-12T03:09:16.515 INFO:teuthology.orchestra.run.smithi177.stdout:(2/30): ceph-selinux-18.2.2-751.g62a11b3c.el9.x 380 kB/s | 25 kB 00:00 2024-03-12T03:09:16.657 INFO:teuthology.orchestra.run.smithi177.stdout:(3/30): libcephfs2-18.2.2-751.g62a11b3c.el9.x86 4.5 MB/s | 659 kB 00:00 2024-03-12T03:09:16.791 INFO:teuthology.orchestra.run.smithi177.stdout:(4/30): libradosstriper1-18.2.2-751.g62a11b3c.e 3.5 MB/s | 474 kB 00:00 2024-03-12T03:09:17.083 INFO:teuthology.orchestra.run.smithi177.stdout:(5/30): ceph-common-18.2.2-751.g62a11b3c.el9.x8 8.1 MB/s | 18 MB 00:02 2024-03-12T03:09:17.183 INFO:teuthology.orchestra.run.smithi177.stdout:(6/30): ceph-base-18.2.2-751.g62a11b3c.el9.x86_ 2.2 MB/s | 5.2 MB 00:02 2024-03-12T03:09:17.208 INFO:teuthology.orchestra.run.smithi177.stdout:(7/30): python3-ceph-argparse-18.2.2-751.g62a11 361 kB/s | 45 kB 00:00 2024-03-12T03:09:17.267 INFO:teuthology.orchestra.run.smithi177.stdout:(8/30): python3-ceph-common-18.2.2-751.g62a11b3 1.5 MB/s | 126 kB 00:00 2024-03-12T03:09:17.292 INFO:teuthology.orchestra.run.smithi177.stdout:(9/30): python3-cephfs-18.2.2-751.g62a11b3c.el9 1.9 MB/s | 161 kB 00:00 2024-03-12T03:09:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:(10/30): librgw2-18.2.2-751.g62a11b3c.el9.x86_6 7.8 MB/s | 4.5 MB 00:00 2024-03-12T03:09:17.384 INFO:teuthology.orchestra.run.smithi177.stdout:(11/30): python3-rbd-18.2.2-751.g62a11b3c.el9.x 3.2 MB/s | 299 kB 00:00 2024-03-12T03:09:17.418 INFO:teuthology.orchestra.run.smithi177.stdout:(12/30): python3-rados-18.2.2-751.g62a11b3c.el9 2.1 MB/s | 321 kB 00:00 2024-03-12T03:09:17.443 INFO:teuthology.orchestra.run.smithi177.stdout:(13/30): python3-rgw-18.2.2-751.g62a11b3c.el9.x 1.2 MB/s | 99 kB 00:00 2024-03-12T03:09:17.576 INFO:teuthology.orchestra.run.smithi177.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2024-03-12T03:09:17.602 INFO:teuthology.orchestra.run.smithi177.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 415 kB/s | 44 kB 00:00 2024-03-12T03:09:17.627 INFO:teuthology.orchestra.run.smithi177.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 761 kB/s | 159 kB 00:00 2024-03-12T03:09:17.677 INFO:teuthology.orchestra.run.smithi177.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-03-12T03:09:17.702 INFO:teuthology.orchestra.run.smithi177.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-12T03:09:17.728 INFO:teuthology.orchestra.run.smithi177.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 411 kB/s | 41 kB 00:00 2024-03-12T03:09:17.903 INFO:teuthology.orchestra.run.smithi177.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 492 kB/s | 111 kB 00:00 2024-03-12T03:09:17.931 INFO:teuthology.orchestra.run.smithi177.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-03-12T03:09:17.955 INFO:teuthology.orchestra.run.smithi177.stdout:(22/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 497 kB/s | 26 kB 00:00 2024-03-12T03:09:17.981 INFO:teuthology.orchestra.run.smithi177.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-03-12T03:09:18.014 INFO:teuthology.orchestra.run.smithi177.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-12T03:09:18.090 INFO:teuthology.orchestra.run.smithi177.stdout:(25/30): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-12T03:09:18.132 INFO:teuthology.orchestra.run.smithi177.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 838 kB 00:00 2024-03-12T03:09:18.157 INFO:teuthology.orchestra.run.smithi177.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-12T03:09:18.200 INFO:teuthology.orchestra.run.smithi177.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-12T03:09:18.791 INFO:teuthology.orchestra.run.smithi177.stdout:(29/30): librbd1-18.2.2-751.g62a11b3c.el9.x86_6 4.7 MB/s | 3.0 MB 00:00 2024-03-12T03:09:20.201 INFO:teuthology.orchestra.run.smithi118.stdout:(1/30): ceph-common-18.2.2-751.g62a11b3c.el9.x8 9.5 MB/s | 18 MB 00:01 2024-03-12T03:09:20.251 INFO:teuthology.orchestra.run.smithi118.stdout:(2/30): ceph-selinux-18.2.2-751.g62a11b3c.el9.x 503 kB/s | 25 kB 00:00 2024-03-12T03:09:20.309 INFO:teuthology.orchestra.run.smithi118.stdout:(3/30): libcephfs2-18.2.2-751.g62a11b3c.el9.x86 11 MB/s | 659 kB 00:00 2024-03-12T03:09:20.368 INFO:teuthology.orchestra.run.smithi118.stdout:(4/30): libradosstriper1-18.2.2-751.g62a11b3c.e 7.9 MB/s | 474 kB 00:00 2024-03-12T03:09:20.619 INFO:teuthology.orchestra.run.smithi118.stdout:(5/30): librgw2-18.2.2-751.g62a11b3c.el9.x86_64 18 MB/s | 4.5 MB 00:00 2024-03-12T03:09:20.660 INFO:teuthology.orchestra.run.smithi118.stdout:(6/30): python3-ceph-argparse-18.2.2-751.g62a11 1.1 MB/s | 45 kB 00:00 2024-03-12T03:09:20.703 INFO:teuthology.orchestra.run.smithi118.stdout:(7/30): python3-ceph-common-18.2.2-751.g62a11b3 2.9 MB/s | 126 kB 00:00 2024-03-12T03:09:20.753 INFO:teuthology.orchestra.run.smithi118.stdout:(8/30): python3-cephfs-18.2.2-751.g62a11b3c.el9 3.1 MB/s | 161 kB 00:00 2024-03-12T03:09:20.803 INFO:teuthology.orchestra.run.smithi118.stdout:(9/30): python3-rados-18.2.2-751.g62a11b3c.el9. 6.3 MB/s | 321 kB 00:00 2024-03-12T03:09:20.853 INFO:teuthology.orchestra.run.smithi118.stdout:(10/30): python3-rbd-18.2.2-751.g62a11b3c.el9.x 5.8 MB/s | 299 kB 00:00 2024-03-12T03:09:20.895 INFO:teuthology.orchestra.run.smithi118.stdout:(11/30): python3-rgw-18.2.2-751.g62a11b3c.el9.x 2.3 MB/s | 99 kB 00:00 2024-03-12T03:09:21.021 INFO:teuthology.orchestra.run.smithi118.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 848 kB/s | 106 kB 00:00 2024-03-12T03:09:21.080 INFO:teuthology.orchestra.run.smithi118.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-12T03:09:21.113 INFO:teuthology.orchestra.run.smithi118.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-12T03:09:21.180 INFO:teuthology.orchestra.run.smithi118.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-12T03:09:21.222 INFO:teuthology.orchestra.run.smithi118.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-12T03:09:21.256 INFO:teuthology.orchestra.run.smithi118.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-12T03:09:21.548 INFO:teuthology.orchestra.run.smithi118.stdout:(18/30): ceph-base-18.2.2-751.g62a11b3c.el9.x86 1.6 MB/s | 5.2 MB 00:03 2024-03-12T03:09:21.815 INFO:teuthology.orchestra.run.smithi177.stdout:(30/30): librados2-18.2.2-751.g62a11b3c.el9.x86 910 kB/s | 3.3 MB 00:03 2024-03-12T03:09:21.820 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:09:21.820 INFO:teuthology.orchestra.run.smithi177.stdout:Total 7.5 MB/s | 53 MB 00:07 2024-03-12T03:09:21.821 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:09:21.857 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:09:21.857 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:09:21.907 INFO:teuthology.orchestra.run.smithi118.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 170 kB/s | 111 kB 00:00 2024-03-12T03:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:09:22.295 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:09:22.532 INFO:teuthology.orchestra.run.smithi118.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 314 kB/s | 309 kB 00:00 2024-03-12T03:09:22.640 INFO:teuthology.orchestra.run.smithi118.stdout:(21/30): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-12T03:09:22.765 INFO:teuthology.orchestra.run.smithi118.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-03-12T03:09:22.916 INFO:teuthology.orchestra.run.smithi118.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 449 kB/s | 67 kB 00:00 2024-03-12T03:09:23.025 INFO:teuthology.orchestra.run.smithi118.stdout:(24/30): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-03-12T03:09:23.142 INFO:teuthology.orchestra.run.smithi118.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-12T03:09:23.217 INFO:teuthology.orchestra.run.smithi118.stdout:(26/30): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-12T03:09:23.317 INFO:teuthology.orchestra.run.smithi118.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-12T03:09:23.584 INFO:teuthology.orchestra.run.smithi118.stdout:(28/30): librbd1-18.2.2-751.g62a11b3c.el9.x86_6 11 MB/s | 3.0 MB 00:00 2024-03-12T03:09:23.680 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:09:23.772 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-12T03:09:23.803 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-12T03:09:23.854 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-12T03:09:24.059 INFO:teuthology.orchestra.run.smithi118.stdout:(29/30): ceph-radosgw-18.2.2-751.g62a11b3c.el9. 1.3 MB/s | 7.7 MB 00:05 2024-03-12T03:09:24.180 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-12T03:09:24.227 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 5/32 2024-03-12T03:09:24.301 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 5/32 2024-03-12T03:09:24.382 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 6/32 2024-03-12T03:09:24.420 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 7/32 2024-03-12T03:09:24.483 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 7/32 2024-03-12T03:09:24.511 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-12T03:09:24.554 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-argparse-2:18.2.2-751.g62a11b3c.el9.x 9/32 2024-03-12T03:09:24.629 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 10/32 2024-03-12T03:09:24.656 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 11/32 2024-03-12T03:09:24.676 INFO:teuthology.orchestra.run.smithi118.stdout:(30/30): librados2-18.2.2-751.g62a11b3c.el9.x86 2.2 MB/s | 3.3 MB 00:01 2024-03-12T03:09:24.681 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:09:24.681 INFO:teuthology.orchestra.run.smithi118.stdout:Total 6.1 MB/s | 53 MB 00:08 2024-03-12T03:09:24.682 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:09:24.717 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:09:24.717 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:09:24.721 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 11/32 2024-03-12T03:09:24.829 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-12T03:09:24.864 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-12T03:09:24.903 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-12T03:09:24.928 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-12T03:09:24.955 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-12T03:09:24.990 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-12T03:09:25.023 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-12T03:09:25.166 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:09:25.166 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:09:25.323 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-12T03:09:25.353 INFO:teuthology.orchestra.run.smithi177.stdout: Upgrading : librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 20/32 2024-03-12T03:09:25.427 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 20/32 2024-03-12T03:09:25.458 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 21/32 2024-03-12T03:09:25.501 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-12T03:09:25.904 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-12T03:09:25.931 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 24/32 2024-03-12T03:09:26.001 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 24/32 2024-03-12T03:09:26.069 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 25/32 2024-03-12T03:09:26.100 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ceph-common-2:18.2.2-751.g62a11b3c.el9.x86 26/32 2024-03-12T03:09:26.569 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:09:26.656 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-03-12T03:09:26.682 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-03-12T03:09:26.732 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-03-12T03:09:27.049 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-03-12T03:09:27.089 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 5/32 2024-03-12T03:09:27.180 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 5/32 2024-03-12T03:09:27.261 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 6/32 2024-03-12T03:09:27.290 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 7/32 2024-03-12T03:09:27.336 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 7/32 2024-03-12T03:09:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-03-12T03:09:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:18.2.2-751.g62a11b3c.el9.x 9/32 2024-03-12T03:09:27.475 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 10/32 2024-03-12T03:09:27.501 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 11/32 2024-03-12T03:09:27.567 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 11/32 2024-03-12T03:09:27.673 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-03-12T03:09:27.700 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/32 2024-03-12T03:09:27.741 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-03-12T03:09:27.773 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-03-12T03:09:27.800 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/32 2024-03-12T03:09:27.836 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-03-12T03:09:27.869 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-03-12T03:09:28.167 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-03-12T03:09:28.199 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 20/32 2024-03-12T03:09:28.221 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 27/32 2024-03-12T03:09:28.255 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 27/32 2024-03-12T03:09:28.273 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 20/32 2024-03-12T03:09:28.304 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 21/32 2024-03-12T03:09:28.342 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-03-12T03:09:28.740 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 23/32 2024-03-12T03:09:28.797 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 24/32 2024-03-12T03:09:28.815 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 27/32 2024-03-12T03:09:28.848 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 24/32 2024-03-12T03:09:28.850 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 28/32 2024-03-12T03:09:28.915 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 25/32 2024-03-12T03:09:28.941 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 28/32 2024-03-12T03:09:28.941 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T03:09:28.941 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T03:09:28.941 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:28.946 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:18.2.2-751.g62a11b3c.el9.x86 26/32 2024-03-12T03:09:28.984 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 29/32 2024-03-12T03:09:30.939 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 27/32 2024-03-12T03:09:30.976 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 27/32 2024-03-12T03:09:31.549 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 27/32 2024-03-12T03:09:31.588 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 28/32 2024-03-12T03:09:31.679 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 28/32 2024-03-12T03:09:31.679 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-12T03:09:31.679 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-12T03:09:31.679 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:31.714 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 29/32 2024-03-12T03:09:38.437 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 29/32 2024-03-12T03:09:38.437 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /sys 2024-03-12T03:09:38.438 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /proc 2024-03-12T03:09:38.438 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /mnt 2024-03-12T03:09:38.438 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /var/tmp 2024-03-12T03:09:38.438 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /home 2024-03-12T03:09:38.438 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /root 2024-03-12T03:09:38.438 INFO:teuthology.orchestra.run.smithi177.stdout:skipping the directory /tmp 2024-03-12T03:09:38.438 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:39.157 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 30/32 2024-03-12T03:09:39.182 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 30/32 2024-03-12T03:09:39.182 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:09:39.182 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T03:09:39.182 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:09:39.183 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:09:39.183 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:39.184 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-12T03:09:39.268 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-12T03:09:39.268 INFO:teuthology.orchestra.run.smithi177.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-12T03:09:40.413 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-12T03:09:40.413 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 1/32 2024-03-12T03:09:40.413 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 2/32 2024-03-12T03:09:40.413 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 3/32 2024-03-12T03:09:40.413 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 4/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 5/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 6/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 7/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-argparse-2:18.2.2-751.g62a11b3c.el9.x 8/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ceph-common-2:18.2.2-751.g62a11b3c.el9.x86 9/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 10/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 11/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 12/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 13/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-12T03:09:40.414 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 29/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-12T03:09:40.415 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 31/32 2024-03-12T03:09:41.548 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-12T03:09:41.548 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:41.548 INFO:teuthology.orchestra.run.smithi177.stdout:Upgraded: 2024-03-12T03:09:41.548 INFO:teuthology.orchestra.run.smithi177.stdout: librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T03:09:41.549 INFO:teuthology.orchestra.run.smithi177.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-argparse-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:41.550 INFO:teuthology.orchestra.run.smithi177.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T03:09:41.551 INFO:teuthology.orchestra.run.smithi177.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T03:09:41.551 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:41.551 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:09:41.799 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-test 2024-03-12T03:09:41.857 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 29/32 2024-03-12T03:09:41.857 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-03-12T03:09:41.858 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-03-12T03:09:41.858 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-03-12T03:09:41.858 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-03-12T03:09:41.858 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-03-12T03:09:41.858 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-03-12T03:09:41.858 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-03-12T03:09:41.858 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:42.358 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:30 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:09:42.516 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 40 M 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T03:09:42.517 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 166 k 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-03-12T03:09:42.518 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:09:42.519 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 41 M 2024-03-12T03:09:42.519 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 168 M 2024-03-12T03:09:42.519 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:09:42.648 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 30/32 2024-03-12T03:09:42.672 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 30/32 2024-03-12T03:09:42.672 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:09:42.672 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-12T03:09:42.672 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:09:42.672 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-12T03:09:42.673 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:42.674 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-12T03:09:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-03-12T03:09:42.751 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 1/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 2/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 3/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 4/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 5/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 6/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 7/32 2024-03-12T03:09:43.950 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:18.2.2-751.g62a11b3c.el9.x 8/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:18.2.2-751.g62a11b3c.el9.x86 9/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 10/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 11/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 12/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 13/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-03-12T03:09:43.951 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 22/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 23/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 26/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 29/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-03-12T03:09:43.952 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 31/32 2024-03-12T03:09:45.054 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-03-12T03:09:45.054 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:45.054 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-03-12T03:09:45.054 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-12T03:09:45.055 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-3.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-12T03:09:45.056 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:45.057 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:09:45.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2024-03-12T03:09:45.957 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:33 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:09:46.111 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:09:46.112 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:09:46.112 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:09:46.112 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:09:46.112 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 40 M 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 166 k 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-03-12T03:09:46.113 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:46.114 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 41 M 2024-03-12T03:09:46.114 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 168 M 2024-03-12T03:09:46.114 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:09:46.572 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-12T03:09:46.656 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-12T03:09:46.697 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-12T03:09:46.764 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-12T03:09:46.806 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): libcephsqlite-18.2.2-751.g62a11b3c.el9.x 388 kB/s | 166 kB 00:00 2024-03-12T03:09:48.857 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-18.2.2-751.g62a11b3c.el9.x86_6 16 MB/s | 40 MB 00:02 2024-03-12T03:09:48.859 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:09:48.860 INFO:teuthology.orchestra.run.smithi118.stdout:Total 15 MB/s | 41 MB 00:02 2024-03-12T03:09:48.873 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:09:48.893 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:09:48.893 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:09:49.267 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:09:49.268 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:09:49.805 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:09:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-12T03:09:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-12T03:09:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:09:50.007 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-12T03:09:50.035 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 5/6 2024-03-12T03:09:51.097 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-12T03:09:51.156 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-12T03:09:51.197 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-12T03:09:51.323 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): libcephsqlite-18.2.2-751.g62a11b3c.el9.x 404 kB/s | 166 kB 00:00 2024-03-12T03:09:51.748 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 340 kB/s | 187 kB 00:00 2024-03-12T03:09:52.932 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): ceph-test-18.2.2-751.g62a11b3c.el9.x86_6 20 MB/s | 40 MB 00:02 2024-03-12T03:09:52.935 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:09:52.935 INFO:teuthology.orchestra.run.smithi177.stdout:Total 3.9 MB/s | 41 MB 00:10 2024-03-12T03:09:52.949 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:09:52.970 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:09:52.970 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:09:53.320 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:09:53.320 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:09:53.834 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 5/6 2024-03-12T03:09:53.859 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:09:53.917 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 6/6 2024-03-12T03:09:53.932 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-12T03:09:53.971 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-12T03:09:54.013 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:09:54.057 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-12T03:09:54.084 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 5/6 2024-03-12T03:09:56.294 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 6/6 2024-03-12T03:09:56.294 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 1/6 2024-03-12T03:09:56.294 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 2/6 2024-03-12T03:09:56.294 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:09:56.294 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-12T03:09:56.294 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-12T03:09:56.712 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-12T03:09:56.712 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:56.712 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:09:56.712 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:56.713 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.6-16.el9.x86_64 2024-03-12T03:09:56.713 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:09:56.713 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T03:09:56.713 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T03:09:56.713 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T03:09:56.713 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:56.713 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:09:57.032 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2024-03-12T03:09:57.600 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:45 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:09:57.752 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:09:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-12T03:09:57.754 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:09:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-12T03:09:57.754 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 6.5 k 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 2.1 M 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 1.4 M 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 245 k 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 4.5 M 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 18 M 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T03:09:57.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T03:09:57.756 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout:Install 33 Packages 2024-03-12T03:09:57.757 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 30 M 2024-03-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 110 M 2024-03-12T03:09:57.758 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:09:57.810 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 5/6 2024-03-12T03:09:57.908 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 6/6 2024-03-12T03:09:58.077 INFO:teuthology.orchestra.run.smithi118.stdout:(1/33): ceph-18.2.2-751.g62a11b3c.el9.x86_64.rp 35 kB/s | 6.5 kB 00:00 2024-03-12T03:09:58.378 INFO:teuthology.orchestra.run.smithi118.stdout:(2/33): ceph-mds-18.2.2-751.g62a11b3c.el9.x86_6 4.3 MB/s | 2.1 MB 00:00 2024-03-12T03:09:58.487 INFO:teuthology.orchestra.run.smithi118.stdout:(3/33): ceph-mon-18.2.2-751.g62a11b3c.el9.x86_6 11 MB/s | 4.5 MB 00:00 2024-03-12T03:09:58.603 INFO:teuthology.orchestra.run.smithi118.stdout:(4/33): ceph-mgr-18.2.2-751.g62a11b3c.el9.x86_6 2.0 MB/s | 1.4 MB 00:00 2024-03-12T03:09:58.720 INFO:teuthology.orchestra.run.smithi118.stdout:(5/33): ceph-mgr-modules-core-18.2.2-751.g62a11 1.0 MB/s | 245 kB 00:00 2024-03-12T03:09:58.737 INFO:teuthology.orchestra.run.smithi118.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-12T03:09:58.762 INFO:teuthology.orchestra.run.smithi118.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-03-12T03:09:58.788 INFO:teuthology.orchestra.run.smithi118.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-12T03:09:58.813 INFO:teuthology.orchestra.run.smithi118.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-12T03:09:58.847 INFO:teuthology.orchestra.run.smithi118.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-12T03:09:58.930 INFO:teuthology.orchestra.run.smithi118.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-03-12T03:09:58.964 INFO:teuthology.orchestra.run.smithi118.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 881 kB/s | 103 kB 00:00 2024-03-12T03:09:58.990 INFO:teuthology.orchestra.run.smithi118.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-12T03:09:59.015 INFO:teuthology.orchestra.run.smithi118.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-12T03:09:59.290 INFO:teuthology.orchestra.run.smithi118.stdout:(15/33): ceph-osd-18.2.2-751.g62a11b3c.el9.x86_ 19 MB/s | 18 MB 00:00 2024-03-12T03:09:59.440 INFO:teuthology.orchestra.run.smithi118.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 102 kB/s | 43 kB 00:00 2024-03-12T03:09:59.633 INFO:teuthology.orchestra.run.smithi118.stdout:(17/33): python3-requests-2.25.1-8.el9.noarch.r 194 kB/s | 125 kB 00:00 2024-03-12T03:09:59.658 INFO:teuthology.orchestra.run.smithi118.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 469 kB/s | 172 kB 00:00 2024-03-12T03:09:59.683 INFO:teuthology.orchestra.run.smithi118.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-12T03:09:59.733 INFO:teuthology.orchestra.run.smithi118.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-12T03:09:59.767 INFO:teuthology.orchestra.run.smithi118.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 398 kB/s | 23 kB 00:00 2024-03-12T03:09:59.792 INFO:teuthology.orchestra.run.smithi118.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 331 kB/s | 19 kB 00:00 2024-03-12T03:09:59.826 INFO:teuthology.orchestra.run.smithi118.stdout:(23/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 55 kB/s | 11 kB 00:00 2024-03-12T03:09:59.842 INFO:teuthology.orchestra.run.smithi118.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-03-12T03:09:59.868 INFO:teuthology.orchestra.run.smithi118.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-03-12T03:09:59.952 INFO:teuthology.orchestra.run.smithi118.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-12T03:09:59.976 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 6/6 2024-03-12T03:09:59.976 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 1/6 2024-03-12T03:09:59.976 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 2/6 2024-03-12T03:09:59.976 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-12T03:09:59.976 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-12T03:09:59.977 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-12T03:09:59.977 INFO:teuthology.orchestra.run.smithi118.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-12T03:10:00.002 INFO:teuthology.orchestra.run.smithi118.stdout:(28/33): python3-more-itertools-8.12.0-2.el9.no 448 kB/s | 79 kB 00:00 2024-03-12T03:10:00.028 INFO:teuthology.orchestra.run.smithi118.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-12T03:10:00.061 INFO:teuthology.orchestra.run.smithi118.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-03-12T03:10:00.137 INFO:teuthology.orchestra.run.smithi118.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-12T03:10:00.162 INFO:teuthology.orchestra.run.smithi118.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-12T03:10:00.187 INFO:teuthology.orchestra.run.smithi118.stdout:(33/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-12T03:10:00.193 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:00.193 INFO:teuthology.orchestra.run.smithi118.stdout:Total 12 MB/s | 30 MB 00:02 2024-03-12T03:10:00.206 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:10:00.246 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:10:00.246 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:10:00.382 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-test-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: jq-1.6-16.el9.x86_64 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: libcephsqlite-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:00.383 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:00.627 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:10:00.627 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:10:00.669 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph 2024-03-12T03:10:01.238 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:00:49 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:01.391 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:01.393 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-12T03:10:01.393 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: ceph x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 6.5 k 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 2.1 M 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 1.4 M 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 245 k 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 4.5 M 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 18 M 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-12T03:10:01.394 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-12T03:10:01.395 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================== 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout:Install 33 Packages 2024-03-12T03:10:01.396 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:01.397 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 30 M 2024-03-12T03:10:01.397 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 110 M 2024-03-12T03:10:01.397 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:10:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:10:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-12T03:10:01.964 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-12T03:10:01.975 INFO:teuthology.orchestra.run.smithi177.stdout:(1/33): ceph-18.2.2-751.g62a11b3c.el9.x86_64.rp 42 kB/s | 6.5 kB 00:00 2024-03-12T03:10:02.005 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-12T03:10:02.104 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-12T03:10:02.199 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-12T03:10:02.241 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-12T03:10:02.241 INFO:teuthology.orchestra.run.smithi177.stdout:(2/33): ceph-mds-18.2.2-751.g62a11b3c.el9.x86_6 5.0 MB/s | 2.1 MB 00:00 2024-03-12T03:10:02.322 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-12T03:10:02.376 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-12T03:10:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-12T03:10:02.450 INFO:teuthology.orchestra.run.smithi177.stdout:(3/33): ceph-mgr-18.2.2-751.g62a11b3c.el9.x86_6 2.3 MB/s | 1.4 MB 00:00 2024-03-12T03:10:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-12T03:10:02.526 INFO:teuthology.orchestra.run.smithi177.stdout:(4/33): ceph-mon-18.2.2-751.g62a11b3c.el9.x86_6 8.1 MB/s | 4.5 MB 00:00 2024-03-12T03:10:02.528 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-12T03:10:02.551 INFO:teuthology.orchestra.run.smithi177.stdout:(5/33): ceph-mgr-modules-core-18.2.2-751.g62a11 2.4 MB/s | 245 kB 00:00 2024-03-12T03:10:02.693 INFO:teuthology.orchestra.run.smithi177.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 866 kB/s | 144 kB 00:00 2024-03-12T03:10:02.712 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-12T03:10:02.718 INFO:teuthology.orchestra.run.smithi177.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 220 kB/s | 32 kB 00:00 2024-03-12T03:10:02.743 INFO:teuthology.orchestra.run.smithi177.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-12T03:10:02.768 INFO:teuthology.orchestra.run.smithi177.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-12T03:10:02.769 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-12T03:10:02.794 INFO:teuthology.orchestra.run.smithi177.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-12T03:10:02.803 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-12T03:10:02.842 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-12T03:10:02.853 INFO:teuthology.orchestra.run.smithi177.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-12T03:10:02.872 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-12T03:10:02.877 INFO:teuthology.orchestra.run.smithi177.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-12T03:10:02.903 INFO:teuthology.orchestra.run.smithi177.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-12T03:10:02.945 INFO:teuthology.orchestra.run.smithi177.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-12T03:10:02.948 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-12T03:10:02.982 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-12T03:10:03.016 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-12T03:10:03.057 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-12T03:10:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-12T03:10:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-12T03:10:03.145 INFO:teuthology.orchestra.run.smithi177.stdout:(15/33): ceph-osd-18.2.2-751.g62a11b3c.el9.x86_ 20 MB/s | 18 MB 00:00 2024-03-12T03:10:03.232 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-12T03:10:03.265 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-12T03:10:03.339 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-12T03:10:03.445 INFO:teuthology.orchestra.run.smithi177.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 87 kB/s | 43 kB 00:00 2024-03-12T03:10:03.484 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-12T03:10:03.487 INFO:teuthology.orchestra.run.smithi177.stdout:(17/33): python3-requests-2.25.1-8.el9.noarch.r 214 kB/s | 125 kB 00:00 2024-03-12T03:10:03.646 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-12T03:10:03.696 INFO:teuthology.orchestra.run.smithi177.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 51 kB/s | 11 kB 00:00 2024-03-12T03:10:03.801 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-751.g62a11b3c.el9.n 28/33 2024-03-12T03:10:03.813 INFO:teuthology.orchestra.run.smithi177.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 257 kB/s | 172 kB 00:00 2024-03-12T03:10:03.829 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 29/33 2024-03-12T03:10:03.830 INFO:teuthology.orchestra.run.smithi177.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-03-12T03:10:03.854 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 29/33 2024-03-12T03:10:03.854 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:03.854 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T03:10:03.854 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:10:03.854 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:10:03.854 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:03.922 INFO:teuthology.orchestra.run.smithi177.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-12T03:10:03.941 INFO:teuthology.orchestra.run.smithi177.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-12T03:10:04.030 INFO:teuthology.orchestra.run.smithi177.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-12T03:10:04.139 INFO:teuthology.orchestra.run.smithi177.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 231 kB/s | 46 kB 00:00 2024-03-12T03:10:04.248 INFO:teuthology.orchestra.run.smithi177.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 363 kB/s | 79 kB 00:00 2024-03-12T03:10:04.290 INFO:teuthology.orchestra.run.smithi177.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 425 kB/s | 358 kB 00:00 2024-03-12T03:10:04.357 INFO:teuthology.orchestra.run.smithi177.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-12T03:10:04.474 INFO:teuthology.orchestra.run.smithi177.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 307 kB/s | 36 kB 00:00 2024-03-12T03:10:04.499 INFO:teuthology.orchestra.run.smithi177.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 431 kB/s | 90 kB 00:00 2024-03-12T03:10:04.757 INFO:teuthology.orchestra.run.smithi177.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 441 kB/s | 272 kB 00:00 2024-03-12T03:10:04.883 INFO:teuthology.orchestra.run.smithi177.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-03-12T03:10:05.041 INFO:teuthology.orchestra.run.smithi177.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 405 kB/s | 230 kB 00:00 2024-03-12T03:10:05.346 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 30/33 2024-03-12T03:10:05.366 INFO:teuthology.orchestra.run.smithi177.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 493 kB/s | 427 kB 00:00 2024-03-12T03:10:05.370 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:05.370 INFO:teuthology.orchestra.run.smithi177.stdout:Total 7.5 MB/s | 30 MB 00:03 2024-03-12T03:10:05.372 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 30/33 2024-03-12T03:10:05.373 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:05.373 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T03:10:05.373 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:10:05.373 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:10:05.373 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:05.384 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:10:05.425 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:10:05.425 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:10:05.796 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:10:05.796 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:10:05.841 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 31/33 2024-03-12T03:10:05.864 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 31/33 2024-03-12T03:10:05.864 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:05.864 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T03:10:05.865 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:10:05.865 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:10:05.865 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:06.073 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 32/33 2024-03-12T03:10:06.096 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 32/33 2024-03-12T03:10:06.096 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:06.096 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T03:10:06.096 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:10:06.096 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:10:06.096 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:06.162 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 33/33 2024-03-12T03:10:07.038 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:10:07.102 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-12T03:10:07.134 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-12T03:10:07.167 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-12T03:10:07.257 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-12T03:10:07.336 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-12T03:10:07.377 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-12T03:10:07.448 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-12T03:10:07.511 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-12T03:10:07.550 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-12T03:10:07.599 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-12T03:10:07.657 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-12T03:10:07.835 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-12T03:10:07.888 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-12T03:10:07.924 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-12T03:10:07.961 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-12T03:10:07.991 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-12T03:10:08.067 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-12T03:10:08.101 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-12T03:10:08.136 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-12T03:10:08.176 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-12T03:10:08.211 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-12T03:10:08.246 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-12T03:10:08.334 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-12T03:10:08.368 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-12T03:10:08.428 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 33/33 2024-03-12T03:10:08.428 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 1/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 2/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 3/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 4/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 5/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-751.g62a11b3c.el9.n 6/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-12T03:10:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-12T03:10:08.430 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-12T03:10:08.431 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-12T03:10:08.431 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-12T03:10:08.431 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-12T03:10:08.431 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-12T03:10:08.431 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-12T03:10:08.439 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-12T03:10:08.575 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-12T03:10:08.694 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-12T03:10:08.845 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-751.g62a11b3c.el9.n 28/33 2024-03-12T03:10:08.882 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 29/33 2024-03-12T03:10:08.907 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 29/33 2024-03-12T03:10:08.907 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:08.907 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-12T03:10:08.907 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:10:08.907 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-12T03:10:08.908 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:09.409 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-12T03:10:09.409 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:09.409 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:10:09.409 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:09.409 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T03:10:09.410 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T03:10:09.411 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T03:10:09.412 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T03:10:09.412 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T03:10:09.412 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T03:10:09.412 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T03:10:09.412 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T03:10:09.412 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:09.412 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:10:09.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2024-03-12T03:10:10.284 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:58 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:10.380 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:10:10.408 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 30/33 2024-03-12T03:10:10.431 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:10:10.432 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:10:10.432 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:10:10.434 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 30/33 2024-03-12T03:10:10.434 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:10.434 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-12T03:10:10.434 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:10:10.434 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-12T03:10:10.434 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:10.496 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2024-03-12T03:10:10.883 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 31/33 2024-03-12T03:10:10.906 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 31/33 2024-03-12T03:10:10.907 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:10.907 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-12T03:10:10.907 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:10:10.907 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-12T03:10:10.907 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:11.049 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:59 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:11.117 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 32/33 2024-03-12T03:10:11.140 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 32/33 2024-03-12T03:10:11.140 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:11.140 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-12T03:10:11.140 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:10:11.140 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-12T03:10:11.140 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:11.195 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 223 k 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:11.196 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 223 k 2024-03-12T03:10:11.197 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 215 k 2024-03-12T03:10:11.197 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:10:11.206 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 33/33 2024-03-12T03:10:11.557 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-18.2.2-751.g62a11b3c.el9.noarch.rpm 621 kB/s | 223 kB 00:00 2024-03-12T03:10:11.557 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:11.558 INFO:teuthology.orchestra.run.smithi118.stdout:Total 619 kB/s | 223 kB 00:00 2024-03-12T03:10:11.558 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:10:11.563 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:10:11.563 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:10:11.565 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:10:11.566 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:10:11.638 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:10:12.075 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:12.159 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:12.569 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:12.889 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:12.890 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:12.890 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:10:12.890 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:12.890 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:12.890 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:10:13.047 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 33/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 1/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 2/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 3/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 4/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 5/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-751.g62a11b3c.el9.n 6/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-12T03:10:13.048 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-12T03:10:13.049 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-12T03:10:13.050 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-12T03:10:13.050 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-12T03:10:13.050 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-12T03:10:13.050 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-12T03:10:13.050 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-12T03:10:13.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2024-03-12T03:10:13.684 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:01 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:13.834 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:10:13.835 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:13.835 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-12T03:10:13.835 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 143 k 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 143 k 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 395 k 2024-03-12T03:10:13.836 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mds-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mon-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-osd-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-12T03:10:13.842 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-12T03:10:13.843 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ply-3.11-14.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:13.844 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:14.143 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-base 2024-03-12T03:10:14.178 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-18.2.2-751.g62a11b3 417 kB/s | 143 kB 00:00 2024-03-12T03:10:14.179 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:14.179 INFO:teuthology.orchestra.run.smithi118.stdout:Total 415 kB/s | 143 kB 00:00 2024-03-12T03:10:14.179 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:10:14.187 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:10:14.187 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:10:14.238 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:10:14.238 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:10:14.379 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:10:14.437 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.e 1/1 2024-03-12T03:10:14.462 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.e 1/1 2024-03-12T03:10:14.462 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:14.462 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T03:10:14.462 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:14.704 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:02 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:14.801 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-base-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:10:14.851 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:10:14.883 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:14.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install cephadm 2024-03-12T03:10:15.417 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.e 1/1 2024-03-12T03:10:15.417 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:15.417 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:10:15.417 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:15.417 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:15.417 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:10:15.467 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:03 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:15.616 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:15.616 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:15.616 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 223 k 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 223 k 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 215 k 2024-03-12T03:10:15.617 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:10:15.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2024-03-12T03:10:15.991 INFO:teuthology.orchestra.run.smithi177.stdout:cephadm-18.2.2-751.g62a11b3c.el9.noarch.rpm 598 kB/s | 223 kB 00:00 2024-03-12T03:10:15.992 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:15.992 INFO:teuthology.orchestra.run.smithi177.stdout:Total 596 kB/s | 223 kB 00:00 2024-03-12T03:10:15.992 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:10:15.997 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:10:15.997 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:10:16.000 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:10:16.000 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:10:16.068 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:10:16.235 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:04 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:16.334 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:10:16.386 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:10:16.387 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:10:16.387 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:10:16.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2024-03-12T03:10:16.494 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:16.587 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:16.998 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:04 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:17.046 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:17.145 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:10:17.145 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:17.145 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 1.7 M 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 23 k 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 15 k 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout:Install 5 Packages 2024-03-12T03:10:17.146 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:17.147 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.0 M 2024-03-12T03:10:17.147 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 8.3 M 2024-03-12T03:10:17.147 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:10:17.384 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/1 2024-03-12T03:10:17.385 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:10:17.385 INFO:teuthology.orchestra.run.smithi177.stdout: cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:17.385 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:17.385 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:17.570 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-immutable-object-cache 2024-03-12T03:10:18.146 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:06 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:18.295 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 143 k 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-12T03:10:18.296 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:18.297 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 143 k 2024-03-12T03:10:18.297 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 395 k 2024-03-12T03:10:18.297 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:10:18.722 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-immutable-object-cache-18.2.2-751.g62a11b3 336 kB/s | 143 kB 00:00 2024-03-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi177.stdout:Total 335 kB/s | 143 kB 00:00 2024-03-12T03:10:18.723 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:10:18.730 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:10:18.730 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:10:18.783 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:10:18.783 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:10:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:10:18.981 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.e 1/1 2024-03-12T03:10:19.004 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.e 1/1 2024-03-12T03:10:19.004 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:10:19.004 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-12T03:10:19.004 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:19.993 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.e 1/1 2024-03-12T03:10:19.993 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:19.993 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:10:19.993 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-immutable-object-cache-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:10:19.993 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:19.993 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:20.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr 2024-03-12T03:10:20.799 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:08 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:20.896 INFO:teuthology.orchestra.run.smithi177.stdout:Package ceph-mgr-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:10:20.947 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:20.948 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:10:20.948 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:21.005 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-dashboard 2024-03-12T03:10:21.567 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:09 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:21.715 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:21.716 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:21.716 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:10:21.716 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:21.716 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:10:21.716 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 1.7 M 2024-03-12T03:10:21.716 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 23 k 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 15 k 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout:Install 5 Packages 2024-03-12T03:10:21.717 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:21.718 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.0 M 2024-03-12T03:10:21.718 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 8.3 M 2024-03-12T03:10:21.718 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:10:29.510 INFO:teuthology.orchestra.run.smithi177.stdout:(1/5): ceph-prometheus-alerts-18.2.2-751.g62a11 21 kB/s | 15 kB 00:00 2024-03-12T03:10:29.533 INFO:teuthology.orchestra.run.smithi177.stdout:(2/5): ceph-grafana-dashboards-18.2.2-751.g62a1 32 kB/s | 23 kB 00:00 2024-03-12T03:10:29.759 INFO:teuthology.orchestra.run.smithi177.stdout:(3/5): ceph-mgr-dashboard-18.2.2-751.g62a11b3c. 1.8 MB/s | 1.7 MB 00:00 2024-03-12T03:10:29.784 INFO:teuthology.orchestra.run.smithi177.stdout:(4/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 112 kB/s | 31 kB 00:00 2024-03-12T03:10:30.576 INFO:teuthology.orchestra.run.smithi177.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 181 kB/s | 188 kB 00:01 2024-03-12T03:10:30.577 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:30.578 INFO:teuthology.orchestra.run.smithi177.stdout:Total 230 kB/s | 2.0 MB 00:08 2024-03-12T03:10:30.578 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:10:30.586 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:10:30.586 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:10:30.657 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:10:30.657 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:10:30.908 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:10:30.990 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-12T03:10:31.022 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-12T03:10:31.050 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-751.g62a11b3c.el9.no 3/5 2024-03-12T03:10:31.284 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-751.g62a11b3c.el9.n 4/5 2024-03-12T03:10:31.320 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 5/5 2024-03-12T03:10:31.799 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 5/5 2024-03-12T03:10:31.799 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-751.g62a11b3c.el9.n 1/5 2024-03-12T03:10:31.799 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 2/5 2024-03-12T03:10:31.799 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-751.g62a11b3c.el9.no 3/5 2024-03-12T03:10:31.799 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-grafana-dashboards-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-prometheus-alerts-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:32.111 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:32.332 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-12T03:10:32.907 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:20 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:33.059 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:33.060 INFO:teuthology.orchestra.run.smithi177.stdout:=========================================================================================== 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout:=========================================================================================== 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 7.4 M 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T03:10:33.061 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout:=========================================================================================== 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout:Install 12 Packages 2024-03-12T03:10:33.062 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:33.063 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 37 M 2024-03-12T03:10:33.063 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 209 M 2024-03-12T03:10:33.063 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:10:33.590 INFO:teuthology.orchestra.run.smithi177.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 217 kB/s | 32 kB 00:00 2024-03-12T03:10:33.624 INFO:teuthology.orchestra.run.smithi177.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 519 kB/s | 17 kB 00:00 2024-03-12T03:10:33.658 INFO:teuthology.orchestra.run.smithi177.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-12T03:10:33.716 INFO:teuthology.orchestra.run.smithi177.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-12T03:10:33.883 INFO:teuthology.orchestra.run.smithi177.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-12T03:10:33.950 INFO:teuthology.orchestra.run.smithi177.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-12T03:10:33.975 INFO:teuthology.orchestra.run.smithi177.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-12T03:10:34.226 INFO:teuthology.orchestra.run.smithi177.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2024-03-12T03:10:34.376 INFO:teuthology.orchestra.run.smithi177.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-12T03:10:34.418 INFO:teuthology.orchestra.run.smithi177.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-12T03:10:34.885 INFO:teuthology.orchestra.run.smithi177.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 483 kB/s | 245 kB 00:00 2024-03-12T03:10:34.993 INFO:teuthology.orchestra.run.smithi177.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.2-7 4.8 MB/s | 7.4 MB 00:01 2024-03-12T03:10:34.997 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:34.997 INFO:teuthology.orchestra.run.smithi177.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-12T03:10:35.037 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:10:35.062 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:10:35.062 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:10:35.502 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:10:35.502 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:10:36.207 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:10:36.295 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-12T03:10:36.322 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-12T03:10:36.417 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:10:36.467 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-12T03:10:36.645 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:10:36.674 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:10:36.743 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-12T03:10:37.423 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-12T03:10:37.520 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-12T03:10:38.788 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-12T03:10:39.823 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-12T03:10:39.849 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b 12/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b 12/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b 1/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:10:42.395 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-12T03:10:42.396 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-12T03:10:42.396 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-12T03:10:42.396 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-12T03:10:42.396 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-12T03:10:42.868 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-12T03:10:42.868 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:42.868 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T03:10:42.869 INFO:teuthology.orchestra.run.smithi177.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T03:10:42.870 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:42.870 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:43.245 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-rook 2024-03-12T03:10:43.810 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:31 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:43.963 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 49 k 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T03:10:43.964 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:43.965 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:10:43.966 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-03-12T03:10:43.966 INFO:teuthology.orchestra.run.smithi177.stdout:Install 13 Packages 2024-03-12T03:10:43.966 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:43.966 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 2.0 M 2024-03-12T03:10:43.966 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 26 M 2024-03-12T03:10:43.967 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:10:44.190 INFO:teuthology.orchestra.run.smithi177.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 165 kB/s | 24 kB 00:00 2024-03-12T03:10:44.207 INFO:teuthology.orchestra.run.smithi177.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-03-12T03:10:44.266 INFO:teuthology.orchestra.run.smithi177.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-12T03:10:44.291 INFO:teuthology.orchestra.run.smithi177.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-12T03:10:44.316 INFO:teuthology.orchestra.run.smithi177.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-12T03:10:44.341 INFO:teuthology.orchestra.run.smithi177.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-12T03:10:44.367 INFO:teuthology.orchestra.run.smithi177.stdout:(7/13): ceph-mgr-rook-18.2.2-751.g62a11b3c.el9. 152 kB/s | 49 kB 00:00 2024-03-12T03:10:44.609 INFO:teuthology.orchestra.run.smithi177.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-12T03:10:44.634 INFO:teuthology.orchestra.run.smithi177.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-12T03:10:44.876 INFO:teuthology.orchestra.run.smithi177.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 398 kB/s | 203 kB 00:00 2024-03-12T03:10:44.976 INFO:teuthology.orchestra.run.smithi177.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-12T03:10:45.243 INFO:teuthology.orchestra.run.smithi177.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 244 kB/s | 90 kB 00:00 2024-03-12T03:10:45.276 INFO:teuthology.orchestra.run.smithi177.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-12T03:10:45.279 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:45.279 INFO:teuthology.orchestra.run.smithi177.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-12T03:10:45.280 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:10:45.294 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:10:45.294 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:10:45.399 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:10:45.400 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:10:45.917 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:10:46.098 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-12T03:10:46.152 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-12T03:10:46.216 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-12T03:10:46.254 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-12T03:10:46.295 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-12T03:10:46.388 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-12T03:10:46.497 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-12T03:10:46.545 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-12T03:10:47.272 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-12T03:10:47.318 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-12T03:10:47.352 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-12T03:10:47.399 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-12T03:10:47.430 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 13/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 13/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 1/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-12T03:10:48.204 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-12T03:10:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-12T03:10:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-12T03:10:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-12T03:10:48.205 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:48.616 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T03:10:48.617 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:48.618 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:10:48.849 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-mgr-cephadm 2024-03-12T03:10:49.417 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:01:37 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:49.568 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:10:49.568 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-03-12T03:10:49.568 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 137 k 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout:======================================================================================= 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout:Install 6 Packages 2024-03-12T03:10:49.569 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:10:49.570 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 6.8 M 2024-03-12T03:10:49.570 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 33 M 2024-03-12T03:10:49.570 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:10:50.971 INFO:teuthology.orchestra.run.smithi118.stdout:(1/5): ceph-prometheus-alerts-18.2.2-751.g62a11 76 kB/s | 15 kB 00:00 2024-03-12T03:10:51.038 INFO:teuthology.orchestra.run.smithi118.stdout:(2/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 462 kB/s | 31 kB 00:00 2024-03-12T03:10:51.063 INFO:teuthology.orchestra.run.smithi118.stdout:(3/5): ceph-grafana-dashboards-18.2.2-751.g62a1 81 kB/s | 23 kB 00:00 2024-03-12T03:10:51.113 INFO:teuthology.orchestra.run.smithi118.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-12T03:10:51.197 INFO:teuthology.orchestra.run.smithi118.stdout:(5/5): ceph-mgr-dashboard-18.2.2-751.g62a11b3c. 4.1 MB/s | 1.7 MB 00:00 2024-03-12T03:10:51.199 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:10:51.200 INFO:teuthology.orchestra.run.smithi118.stdout:Total 60 kB/s | 2.0 MB 00:34 2024-03-12T03:10:51.200 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:10:51.207 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:10:51.207 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:10:51.271 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:10:51.271 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:10:51.517 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:10:51.594 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-03-12T03:10:51.626 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-03-12T03:10:51.655 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-751.g62a11b3c.el9.no 3/5 2024-03-12T03:10:51.886 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-751.g62a11b3c.el9.n 4/5 2024-03-12T03:10:51.925 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 5/5 2024-03-12T03:10:52.454 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 5/5 2024-03-12T03:10:52.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-751.g62a11b3c.el9.n 1/5 2024-03-12T03:10:52.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 2/5 2024-03-12T03:10:52.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-751.g62a11b3c.el9.no 3/5 2024-03-12T03:10:52.454 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:52.755 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:10:52.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-12T03:10:53.540 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:41 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:10:53.690 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout:=========================================================================================== 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout:=========================================================================================== 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 7.4 M 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-12T03:10:53.691 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout:=========================================================================================== 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-03-12T03:10:53.692 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:10:53.693 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 37 M 2024-03-12T03:10:53.693 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 209 M 2024-03-12T03:10:53.693 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:13:01.007 INFO:teuthology.orchestra.run.smithi177.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-12T03:13:01.031 INFO:teuthology.orchestra.run.smithi177.stdout:(2/6): ceph-mgr-cephadm-18.2.2-751.g62a11b3c.el 362 kB/s | 137 kB 00:00 2024-03-12T03:13:01.489 INFO:teuthology.orchestra.run.smithi177.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 296 kB/s | 248 kB 00:00 2024-03-12T03:13:01.681 INFO:teuthology.orchestra.run.smithi177.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 89 kB/s | 58 kB 00:00 2024-03-12T03:13:02.165 INFO:teuthology.orchestra.run.smithi177.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 75 kB/s | 51 kB 00:00 2024-03-12T03:13:02.565 INFO:teuthology.orchestra.run.smithi177.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 351 kB/s | 548 kB 00:01 2024-03-12T03:13:02.566 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:02.566 INFO:teuthology.orchestra.run.smithi177.stdout:Total 52 kB/s | 6.8 MB 02:12 2024-03-12T03:13:02.579 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:13:02.591 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:13:02.591 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:13:02.687 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:13:02.688 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:13:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:13:03.146 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-12T03:13:03.199 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-12T03:13:03.477 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 188 kB/s | 32 kB 00:00 2024-03-12T03:13:03.511 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2024-03-12T03:13:03.553 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 828 kB/s | 34 kB 00:00 2024-03-12T03:13:03.612 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-03-12T03:13:03.709 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-12T03:13:03.770 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-12T03:13:03.800 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-12T03:13:03.854 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-12T03:13:03.862 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-12T03:13:03.879 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-12T03:13:03.886 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 6/6 2024-03-12T03:13:04.129 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2024-03-12T03:13:04.279 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): ceph-mgr-diskprediction-local-18.2.2-75 7.6 MB/s | 7.4 MB 00:00 2024-03-12T03:13:04.363 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-12T03:13:04.413 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 689 kB/s | 195 kB 00:00 2024-03-12T03:13:04.530 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 980 kB/s | 245 kB 00:00 2024-03-12T03:13:04.532 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:04.533 INFO:teuthology.orchestra.run.smithi118.stdout:Total 290 kB/s | 37 MB 02:10 2024-03-12T03:13:04.570 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:13:04.594 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:13:04.594 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:13:04.633 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 6/6 2024-03-12T03:13:04.633 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/6 2024-03-12T03:13:04.634 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-12T03:13:04.634 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-12T03:13:04.634 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-12T03:13:04.634 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-12T03:13:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-12T03:13:04.949 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:04.949 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:04.950 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:05.036 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:13:05.036 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:13:05.222 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install ceph-fuse 2024-03-12T03:13:05.749 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:13:05.792 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:53 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:05.836 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-12T03:13:05.868 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-12T03:13:05.943 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:05.943 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:05.943 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repository Size 2024-03-12T03:13:05.943 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:05.943 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:13:05.943 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 841 k 2024-03-12T03:13:05.943 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout:Install 2 Packages 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 920 k 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 2.7 M 2024-03-12T03:13:05.944 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:13:05.961 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:13:06.005 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-12T03:13:06.100 INFO:teuthology.orchestra.run.smithi177.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 508 kB/s | 79 kB 00:00 2024-03-12T03:13:06.175 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:13:06.204 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:13:06.258 INFO:teuthology.orchestra.run.smithi177.stdout:(2/2): ceph-fuse-18.2.2-751.g62a11b3c.el9.x86_6 2.6 MB/s | 841 kB 00:00 2024-03-12T03:13:06.259 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:06.260 INFO:teuthology.orchestra.run.smithi177.stdout:Total 2.8 MB/s | 920 kB 00:00 2024-03-12T03:13:06.260 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:13:06.280 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:13:06.280 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:13:06.281 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-12T03:13:06.356 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:13:06.356 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:13:06.547 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:13:06.690 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-12T03:13:06.782 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 2/2 2024-03-12T03:13:06.936 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-12T03:13:07.049 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-12T03:13:07.515 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 2/2 2024-03-12T03:13:07.515 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/2 2024-03-12T03:13:07.838 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-12T03:13:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:07.839 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:13:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-12T03:13:07.839 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:07.839 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:08.089 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados-devel 2024-03-12T03:13:08.296 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-12T03:13:08.672 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:56 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:08.821 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 127 k 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:08.822 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 127 k 2024-03-12T03:13:08.823 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 456 k 2024-03-12T03:13:08.823 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:13:09.137 INFO:teuthology.orchestra.run.smithi177.stdout:librados-devel-18.2.2-751.g62a11b3c.el9.x86_64. 402 kB/s | 127 kB 00:00 2024-03-12T03:13:09.138 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:09.138 INFO:teuthology.orchestra.run.smithi177.stdout:Total 401 kB/s | 127 kB 00:00 2024-03-12T03:13:09.138 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:13:09.144 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:13:09.144 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:13:09.199 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:13:09.199 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:13:09.340 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-12T03:13:09.350 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:13:09.363 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b 12/12 2024-03-12T03:13:09.461 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:09.886 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:10.187 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:10.188 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:13:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:13:10.188 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:10.188 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:10.433 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs2 2024-03-12T03:13:11.005 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:59 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:11.103 INFO:teuthology.orchestra.run.smithi177.stdout:Package libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:13:11.158 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:11.160 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:13:11.160 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:11.214 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install libcephfs-devel 2024-03-12T03:13:11.780 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:03:59 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:11.895 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b 12/12 2024-03-12T03:13:11.895 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b 1/12 2024-03-12T03:13:11.895 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-12T03:13:11.895 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-12T03:13:11.895 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-12T03:13:11.895 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-12T03:13:11.896 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-12T03:13:11.896 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-12T03:13:11.896 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-12T03:13:11.896 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-12T03:13:11.896 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-12T03:13:11.896 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-12T03:13:11.930 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:11.930 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 31 k 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-12T03:13:11.931 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:11.932 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 31 k 2024-03-12T03:13:11.932 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 139 k 2024-03-12T03:13:11.932 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:13:12.174 INFO:teuthology.orchestra.run.smithi177.stdout:libcephfs-devel-18.2.2-751.g62a11b3c.el9.x86_64 128 kB/s | 31 kB 00:00 2024-03-12T03:13:12.175 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:12.175 INFO:teuthology.orchestra.run.smithi177.stdout:Total 127 kB/s | 31 kB 00:00 2024-03-12T03:13:12.175 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:13:12.178 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:13:12.178 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:13:12.210 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:13:12.211 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:13:12.298 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:13:12.342 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-12T03:13:12.342 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:12.342 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:13:12.342 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:13:12.342 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-12T03:13:12.342 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-12T03:13:12.342 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:12.343 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:13:12.413 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:12.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2024-03-12T03:13:12.801 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:13.068 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:13.068 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:13.068 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:13:13.068 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:13:13.068 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:13.068 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:13.239 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librados2 2024-03-12T03:13:13.283 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:01 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:13.434 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:13:13.435 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-03-12T03:13:13.435 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:13:13.435 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-03-12T03:13:13.435 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:13:13.435 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 49 k 2024-03-12T03:13:13.435 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:13:13.435 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:13:13.436 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-03-12T03:13:13.437 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2024-03-12T03:13:13.437 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:13.437 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.0 M 2024-03-12T03:13:13.437 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2024-03-12T03:13:13.437 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:13:13.805 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:01 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:13.902 INFO:teuthology.orchestra.run.smithi177.stdout:Package librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:13:13.940 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-03-12T03:13:13.956 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:13.957 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:13:13.957 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:13.965 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 149 kB/s | 24 kB 00:00 2024-03-12T03:13:14.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install librbd1 2024-03-12T03:13:14.025 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-12T03:13:14.049 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-12T03:13:14.074 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-12T03:13:14.099 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-12T03:13:14.125 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): ceph-mgr-rook-18.2.2-751.g62a11b3c.el9. 152 kB/s | 49 kB 00:00 2024-03-12T03:13:14.400 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-12T03:13:14.426 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2024-03-12T03:13:14.589 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:02 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:14.643 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2024-03-12T03:13:14.687 INFO:teuthology.orchestra.run.smithi177.stdout:Package librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:13:14.740 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:14.741 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:13:14.741 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:14.800 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rados 2024-03-12T03:13:14.851 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 431 kB/s | 90 kB 00:00 2024-03-12T03:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-google-auth-2.28.0-1.el9.noarc 243 kB/s | 203 kB 00:00 2024-03-12T03:13:15.369 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:03 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:15.478 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:13:15.534 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:15.535 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:13:15.535 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:15.590 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rgw 2024-03-12T03:13:16.154 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:04 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:16.251 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:13:16.304 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:16.305 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:13:16.305 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:16.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-cephfs 2024-03-12T03:13:16.451 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 510 kB/s | 1.0 MB 00:02 2024-03-12T03:13:16.453 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:16.454 INFO:teuthology.orchestra.run.smithi118.stdout:Total 694 kB/s | 2.0 MB 00:03 2024-03-12T03:13:16.454 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:13:16.469 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:13:16.469 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:13:16.574 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:13:16.574 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:13:16.930 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:04 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:17.029 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:13:17.081 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:17.082 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:13:17.082 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:17.108 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:13:17.140 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install python3-rbd 2024-03-12T03:13:17.283 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-12T03:13:17.344 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-12T03:13:17.400 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-12T03:13:17.437 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-12T03:13:17.479 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-12T03:13:17.573 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-12T03:13:17.682 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-12T03:13:17.706 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:05 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:17.739 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-12T03:13:17.804 INFO:teuthology.orchestra.run.smithi177.stdout:Package python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:13:17.856 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:17.857 INFO:teuthology.orchestra.run.smithi177.stdout:Nothing to do. 2024-03-12T03:13:17.857 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:17.913 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-fuse 2024-03-12T03:13:18.458 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-12T03:13:18.473 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:06 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:18.494 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-12T03:13:18.528 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-12T03:13:18.577 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-12T03:13:18.599 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 13/13 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 87 k 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:13:18.623 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:18.624 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-12T03:13:18.624 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:18.624 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 87 k 2024-03-12T03:13:18.624 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 227 k 2024-03-12T03:13:18.624 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:13:18.969 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-fuse-18.2.2-751.g62a11b3c.el9.x86_64.rpm 252 kB/s | 87 kB 00:00 2024-03-12T03:13:18.970 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:18.970 INFO:teuthology.orchestra.run.smithi177.stdout:Total 251 kB/s | 87 kB 00:00 2024-03-12T03:13:18.970 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:13:18.976 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:13:18.976 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:13:19.031 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:13:19.031 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:13:19.191 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:13:19.291 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:19.509 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 13/13 2024-03-12T03:13:19.510 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 1/13 2024-03-12T03:13:19.510 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-12T03:13:19.510 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-12T03:13:19.510 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-12T03:13:19.510 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-12T03:13:19.511 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-12T03:13:19.511 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-12T03:13:19.511 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-12T03:13:19.511 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-12T03:13:19.511 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-12T03:13:19.511 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-12T03:13:19.512 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-12T03:13:19.703 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-12T03:13:19.889 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-12T03:13:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-12T03:13:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-12T03:13:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-12T03:13:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-12T03:13:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-12T03:13:19.890 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:19.890 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:13:19.994 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:19.995 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:19.995 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:13:19.995 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:13:19.995 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:19.995 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:20.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2024-03-12T03:13:20.190 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-mirror 2024-03-12T03:13:20.695 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:08 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:20.762 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:08 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:20.846 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:13:20.846 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:18.2.2-751.g62a11b3c.el9 ceph-noarch 137 k 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:20.847 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:13:20.848 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================= 2024-03-12T03:13:20.848 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-03-12T03:13:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:20.848 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 6.8 M 2024-03-12T03:13:20.848 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 33 M 2024-03-12T03:13:20.848 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:13:20.916 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:20.916 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:20.916 INFO:teuthology.orchestra.run.smithi177.stdout: Package Arch Version Repo Size 2024-03-12T03:13:20.916 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 3.0 M 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 3.0 M 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 12 M 2024-03-12T03:13:20.917 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:13:21.330 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-mirror-18.2.2-751.g62a11b3c.el9.x86_64.rpm 7.2 MB/s | 3.0 MB 00:00 2024-03-12T03:13:21.331 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:21.331 INFO:teuthology.orchestra.run.smithi177.stdout:Total 7.2 MB/s | 3.0 MB 00:00 2024-03-12T03:13:21.331 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:13:21.340 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:13:21.340 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:13:21.411 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:13:21.412 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:13:21.821 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:13:21.881 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:21.904 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:21.904 INFO:teuthology.orchestra.run.smithi177.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:13:21.905 INFO:teuthology.orchestra.run.smithi177.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T03:13:21.905 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:13:21.905 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:13:21.905 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:22.371 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-12T03:13:22.522 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-12T03:13:22.581 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): ceph-mgr-cephadm-18.2.2-751.g62a11b3c.el 359 kB/s | 137 kB 00:00 2024-03-12T03:13:22.731 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 276 kB/s | 58 kB 00:00 2024-03-12T03:13:23.044 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:23.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:23.044 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:13:23.044 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:13:23.044 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:23.044 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:23.291 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum -y install rbd-nbd 2024-03-12T03:13:23.863 INFO:teuthology.orchestra.run.smithi177.stdout:Last metadata expiration check: 0:04:11 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:24.015 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-12T03:13:24.016 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:24.016 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-12T03:13:24.016 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 172 k 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout:Install 1 Package 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 172 k 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 494 k 2024-03-12T03:13:24.017 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-12T03:13:24.258 INFO:teuthology.orchestra.run.smithi177.stdout:rbd-nbd-18.2.2-751.g62a11b3c.el9.x86_64.rpm 714 kB/s | 172 kB 00:00 2024-03-12T03:13:24.259 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:24.259 INFO:teuthology.orchestra.run.smithi177.stdout:Total 710 kB/s | 172 kB 00:00 2024-03-12T03:13:24.259 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-12T03:13:24.267 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-12T03:13:24.267 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-12T03:13:24.323 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-12T03:13:24.323 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-12T03:13:24.398 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 270 kB/s | 548 kB 00:02 2024-03-12T03:13:24.476 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-12T03:13:24.590 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:25.021 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:25.321 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:13:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:25.322 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-12T03:13:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:13:25.322 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:13:25.322 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-12T03:13:25.579 DEBUG:teuthology.parallel:result is None 2024-03-12T03:13:54.519 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 1.6 kB/s | 51 kB 00:31 2024-03-12T03:13:54.520 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:54.521 INFO:teuthology.orchestra.run.smithi118.stdout:Total 207 kB/s | 6.8 MB 00:33 2024-03-12T03:13:54.534 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:13:54.546 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:13:54.639 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:13:54.640 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:13:54.947 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:13:55.119 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-12T03:13:55.162 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-12T03:13:55.682 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-12T03:13:55.781 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-12T03:13:55.842 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-12T03:13:55.866 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 6/6 2024-03-12T03:13:56.607 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 6/6 2024-03-12T03:13:56.607 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 1/6 2024-03-12T03:13:56.607 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-12T03:13:56.607 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-12T03:13:56.607 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-12T03:13:56.607 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-12T03:13:56.926 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-12T03:13:56.926 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:56.926 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:13:56.926 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:18.2.2-751.g62a11b3c.el9.noarch 2024-03-12T03:13:56.926 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-12T03:13:56.926 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-12T03:13:56.926 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-12T03:13:56.927 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-12T03:13:56.927 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-12T03:13:56.927 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:56.927 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:13:57.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2024-03-12T03:13:57.776 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:45 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:13:57.926 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:13:57.926 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:13:57.926 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 841 k 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-03-12T03:13:57.927 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:57.928 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 920 k 2024-03-12T03:13:57.928 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 2.7 M 2024-03-12T03:13:57.928 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:13:58.071 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 550 kB/s | 79 kB 00:00 2024-03-12T03:13:58.213 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-fuse-18.2.2-751.g62a11b3c.el9.x86_6 2.9 MB/s | 841 kB 00:00 2024-03-12T03:13:58.214 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:13:58.214 INFO:teuthology.orchestra.run.smithi118.stdout:Total 3.1 MB/s | 920 kB 00:00 2024-03-12T03:13:58.214 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:13:58.235 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:13:58.235 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:13:58.310 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:13:58.310 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:13:58.488 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:13:58.627 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-12T03:13:58.696 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 2/2 2024-03-12T03:13:59.423 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 2/2 2024-03-12T03:13:59.423 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/2 2024-03-12T03:13:59.752 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-12T03:13:59.752 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:59.752 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:13:59.752 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-12T03:13:59.752 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:13:59.752 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:00.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2024-03-12T03:14:00.585 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:48 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 127 k 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-12T03:14:00.743 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 127 k 2024-03-12T03:14:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 456 k 2024-03-12T03:14:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:14:00.971 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-18.2.2-751.g62a11b3c.el9.x86_64. 556 kB/s | 127 kB 00:00 2024-03-12T03:14:00.971 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:14:00.972 INFO:teuthology.orchestra.run.smithi118.stdout:Total 552 kB/s | 127 kB 00:00 2024-03-12T03:14:00.972 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:14:00.977 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:14:00.977 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:14:01.033 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:14:01.033 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:14:01.174 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:14:01.278 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:01.750 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:02.064 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:02.065 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:14:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:14:02.065 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:02.065 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:02.348 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2024-03-12T03:14:02.920 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:50 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:03.021 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:14:03.076 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:03.078 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:14:03.078 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:03.151 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2024-03-12T03:14:03.718 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:51 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:03.867 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:03.868 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:03.868 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-12T03:14:03.868 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 31 k 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 31 k 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 139 k 2024-03-12T03:14:03.869 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:14:04.117 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-18.2.2-751.g62a11b3c.el9.x86_64 125 kB/s | 31 kB 00:00 2024-03-12T03:14:04.117 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:14:04.118 INFO:teuthology.orchestra.run.smithi118.stdout:Total 125 kB/s | 31 kB 00:00 2024-03-12T03:14:04.118 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:14:04.121 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:14:04.121 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:14:04.149 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:14:04.149 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:14:04.250 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:14:04.348 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:04.829 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:05.093 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:05.093 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:05.093 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:14:05.093 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:14:05.093 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:05.093 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:05.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2024-03-12T03:14:05.853 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:53 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:05.950 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:14:06.002 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:06.003 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:14:06.004 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:06.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2024-03-12T03:14:06.639 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:54 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:06.737 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:14:06.789 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:06.790 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:14:06.790 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:06.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2024-03-12T03:14:07.412 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:55 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:07.509 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:14:07.562 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:07.563 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:14:07.563 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:07.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2024-03-12T03:14:08.201 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:56 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:08.299 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:14:08.353 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:08.354 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:14:08.354 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:08.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2024-03-12T03:14:08.983 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:56 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:09.080 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:14:09.132 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:09.133 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:14:09.134 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:09.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2024-03-12T03:14:09.758 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:57 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:09.855 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:18.2.2-751.g62a11b3c.el9.x86_64 is already installed. 2024-03-12T03:14:09.907 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:09.908 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-12T03:14:09.908 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:09.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2024-03-12T03:14:10.531 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:04:58 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:10.679 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:10.680 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-12T03:14:10.680 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:10.680 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:14:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 87 k 2024-03-12T03:14:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:10.681 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:14:10.681 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:10.681 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-12T03:14:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:10.681 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 87 k 2024-03-12T03:14:10.681 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 227 k 2024-03-12T03:14:10.681 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:14:10.895 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-18.2.2-751.g62a11b3c.el9.x86_64.rpm 405 kB/s | 87 kB 00:00 2024-03-12T03:14:10.895 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:14:10.896 INFO:teuthology.orchestra.run.smithi118.stdout:Total 402 kB/s | 87 kB 00:00 2024-03-12T03:14:10.896 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:14:10.902 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:14:10.903 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:14:10.956 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:14:10.957 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:14:11.133 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:14:11.226 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:11.669 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:11.970 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:11.971 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:11.971 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:14:11.971 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:14:11.971 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:11.971 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:12.203 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2024-03-12T03:14:12.768 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:05:00 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:12.925 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:12.926 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:12.926 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-12T03:14:12.926 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:12.926 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:14:12.926 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 3.0 M 2024-03-12T03:14:12.926 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:12.927 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:14:12.927 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:12.927 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-12T03:14:12.927 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:12.927 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.0 M 2024-03-12T03:14:12.927 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 12 M 2024-03-12T03:14:12.927 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:14:13.541 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-18.2.2-751.g62a11b3c.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-12T03:14:13.541 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:14:13.542 INFO:teuthology.orchestra.run.smithi118.stdout:Total 4.9 MB/s | 3.0 MB 00:00 2024-03-12T03:14:13.542 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:14:13.550 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:14:13.550 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:14:13.622 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:14:13.622 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:14:14.031 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:14:14.091 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:14.115 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:14.116 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-12T03:14:14.116 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-12T03:14:14.116 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:14:14.116 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-12T03:14:14.116 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:15.208 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:15.208 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:15.208 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:14:15.208 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:14:15.208 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:15.208 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:15.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2024-03-12T03:14:16.036 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:05:04 ago on Tue 12 Mar 2024 03:09:12 AM UTC. 2024-03-12T03:14:16.186 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-12T03:14:16.186 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:18.2.2-751.g62a11b3c.el9 ceph 172 k 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 172 k 2024-03-12T03:14:16.187 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 494 k 2024-03-12T03:14:16.188 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-12T03:14:16.412 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-18.2.2-751.g62a11b3c.el9.x86_64.rpm 769 kB/s | 172 kB 00:00 2024-03-12T03:14:16.413 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:14:16.413 INFO:teuthology.orchestra.run.smithi118.stdout:Total 764 kB/s | 172 kB 00:00 2024-03-12T03:14:16.413 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-12T03:14:16.421 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-12T03:14:16.421 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-12T03:14:16.476 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-12T03:14:16.476 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-12T03:14:16.619 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-12T03:14:16.726 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:17.252 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:17.583 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 1/1 2024-03-12T03:14:17.583 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:17.584 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-12T03:14:17.584 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:18.2.2-751.g62a11b3c.el9.x86_64 2024-03-12T03:14:17.584 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:14:17.584 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-12T03:14:17.853 DEBUG:teuthology.parallel:result is None 2024-03-12T03:14:17.853 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:14:17.854 INFO:teuthology.packaging:ref: None 2024-03-12T03:14:17.854 INFO:teuthology.packaging:tag: None 2024-03-12T03:14:17.854 INFO:teuthology.packaging:branch: reef 2024-03-12T03:14:17.854 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:14:17.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-12T03:14:18.093 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T03:14:18.113 INFO:teuthology.orchestra.run.smithi118.stdout:18.2.2-751.g62a11b3c.el9 2024-03-12T03:14:18.114 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-751.g62a11b3c.el9 2024-03-12T03:14:18.114 INFO:teuthology.task.install:The correct ceph version 18.2.2-751.g62a11b3c is installed. 2024-03-12T03:14:18.116 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:14:18.116 INFO:teuthology.packaging:ref: None 2024-03-12T03:14:18.116 INFO:teuthology.packaging:tag: None 2024-03-12T03:14:18.116 INFO:teuthology.packaging:branch: reef 2024-03-12T03:14:18.116 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:14:18.117 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-03-12T03:14:18.430 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-12T03:14:18.455 INFO:teuthology.orchestra.run.smithi177.stdout:18.2.2-751.g62a11b3c.el9 2024-03-12T03:14:18.455 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-751.g62a11b3c.el9 2024-03-12T03:14:18.455 INFO:teuthology.task.install:The correct ceph version 18.2.2-751.g62a11b3c is installed. 2024-03-12T03:14:18.457 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-12T03:14:18.457 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:14:18.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T03:14:18.485 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:14:18.485 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-12T03:14:18.526 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-12T03:14:18.527 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:14:18.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T03:14:18.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T03:14:18.622 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:14:18.622 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-03-12T03:14:18.652 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-12T03:14:18.722 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-12T03:14:18.722 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:14:18.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T03:14:18.749 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T03:14:18.816 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:14:18.816 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-12T03:14:18.843 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-12T03:14:18.910 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-12T03:14:18.910 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:14:18.910 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T03:14:18.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T03:14:19.003 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:14:19.004 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-03-12T03:14:19.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-12T03:14:19.098 INFO:teuthology.run_tasks:Running task print... 2024-03-12T03:14:19.110 INFO:teuthology.task.print:**** done install task... 2024-03-12T03:14:19.110 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T03:14:19.205 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-12T03:14:19.206 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-03-12T03:14:19.206 INFO:tasks.cephadm:Cluster fsid is 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:14:19.206 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T03:14:19.206 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-12T03:14:19.206 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi118': '172.21.15.118', 'mon.smithi177': '172.21.15.177'} 2024-03-12T03:14:19.206 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T03:14:19.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo hostname $(hostname -s) 2024-03-12T03:14:19.234 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-03-12T03:14:19.262 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-03-12T03:14:19.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:14:19.403 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979.g2b0101eb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-11 00:47:00.272739', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-12T03:14:19.626 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref reef, sha1 62a11b3cb13eae2a420414a5cf4f907b6bc88089 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-03-12T03:14:19.628 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/x86_64/flavors/default/cephadm 2024-03-12T03:14:19.628 DEBUG:teuthology.orchestra.run.smithi118:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:14:20.171 INFO:teuthology.orchestra.run.smithi118.stdout:-rw-r--r--. 1 ubuntu ubuntu 214739 Mar 12 03:14 /home/ubuntu/cephtest/cephadm 2024-03-12T03:14:20.171 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/reef/62a11b3cb13eae2a420414a5cf4f907b6bc88089/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:14:20.979 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 214739 Mar 12 03:14 /home/ubuntu/cephtest/cephadm 2024-03-12T03:14:20.980 DEBUG:teuthology.orchestra.run.smithi118:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:14:20.999 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:14:21.036 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-03-12T03:14:21.037 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-12T03:14:21.042 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-03-12T03:14:21.842 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-12T03:14:21.949 INFO:teuthology.orchestra.run.smithi118.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-12T03:15:14.219 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-12T03:15:14.220 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)", 2024-03-12T03:15:14.220 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464", 2024-03-12T03:15:14.220 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-03-12T03:15:14.220 INFO:teuthology.orchestra.run.smithi177.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6" 2024-03-12T03:15:14.220 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-03-12T03:15:14.220 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-12T03:15:14.507 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-12T03:15:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_version": "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)", 2024-03-12T03:15:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "image_id": "fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464", 2024-03-12T03:15:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "repo_digests": [ 2024-03-12T03:15:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6" 2024-03-12T03:15:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-12T03:15:14.508 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-12T03:15:14.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph 2024-03-12T03:15:14.586 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-03-12T03:15:14.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /etc/ceph 2024-03-12T03:15:14.642 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-03-12T03:15:14.683 INFO:tasks.cephadm:Writing seed config... 2024-03-12T03:15:14.684 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-12T03:15:14.684 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-12T03:15:14.684 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-12T03:15:14.684 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-12T03:15:14.684 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-12T03:15:14.684 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-12T03:15:14.684 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-12T03:15:14.685 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T03:15:14.686 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T03:15:14.687 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T03:15:14.687 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-12T03:15:14.687 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:15:14.688 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T03:15:14.703 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9d8e7844-e01e-11ee-95c8-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-12T03:15:14.703 DEBUG:teuthology.orchestra.run.smithi118:mon.smithi118> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118.service 2024-03-12T03:15:14.745 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T03:15:14.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.118 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:15:14.901 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:15:14.901 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '9d8e7844-e01e-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.118', '--skip-admin-label'] 2024-03-12T03:15:14.926 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:15:14.927 INFO:teuthology.orchestra.run.smithi118.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-12T03:15:14.927 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying podman|docker is present... 2024-03-12T03:15:14.949 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:15:14.949 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying lvm2 is present... 2024-03-12T03:15:14.949 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying time synchronization is in place... 2024-03-12T03:15:14.956 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:15:14.956 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:15:14.962 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:15:14.962 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-12T03:15:14.968 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout enabled 2024-03-12T03:15:14.974 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout active 2024-03-12T03:15:14.974 INFO:teuthology.orchestra.run.smithi118.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:15:14.974 INFO:teuthology.orchestra.run.smithi118.stdout:Repeating the final host check... 2024-03-12T03:15:14.996 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-12T03:15:14.997 INFO:teuthology.orchestra.run.smithi118.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-12T03:15:14.997 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl is present 2024-03-12T03:15:14.997 INFO:teuthology.orchestra.run.smithi118.stdout:lvcreate is present 2024-03-12T03:15:15.003 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-12T03:15:15.003 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:15:15.008 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-12T03:15:15.009 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-12T03:15:15.014 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout enabled 2024-03-12T03:15:15.021 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout active 2024-03-12T03:15:15.021 INFO:teuthology.orchestra.run.smithi118.stdout:Unit chronyd.service is enabled and running 2024-03-12T03:15:15.021 INFO:teuthology.orchestra.run.smithi118.stdout:Host looks OK 2024-03-12T03:15:15.021 INFO:teuthology.orchestra.run.smithi118.stdout:Cluster fsid: 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:15.021 INFO:teuthology.orchestra.run.smithi118.stdout:Acquiring lock 140387493349600 on /run/cephadm/9d8e7844-e01e-11ee-95c8-87774f69a715.lock 2024-03-12T03:15:15.021 INFO:teuthology.orchestra.run.smithi118.stdout:Lock 140387493349600 acquired on /run/cephadm/9d8e7844-e01e-11ee-95c8-87774f69a715.lock 2024-03-12T03:15:15.022 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying IP 172.21.15.118 port 3300 ... 2024-03-12T03:15:15.022 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying IP 172.21.15.118 port 6789 ... 2024-03-12T03:15:15.022 INFO:teuthology.orchestra.run.smithi118.stdout:Base mon IP(s) is [172.21.15.118:3300, 172.21.15.118:6789], mon addrv is [v2:172.21.15.118:3300,v1:172.21.15.118:6789] 2024-03-12T03:15:15.025 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.118 metric 100 2024-03-12T03:15:15.025 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.118 metric 100 2024-03-12T03:15:15.027 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T03:15:15.027 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-12T03:15:15.028 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2024-03-12T03:15:15.029 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T03:15:15.030 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-12T03:15:15.030 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:15:15.030 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-12T03:15:15.030 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d725/64 scope link 2024-03-12T03:15:15.030 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-12T03:15:15.030 INFO:teuthology.orchestra.run.smithi118.stdout:Mon IP `172.21.15.118` is in CIDR network `172.21.0.0/20` 2024-03-12T03:15:15.031 INFO:teuthology.orchestra.run.smithi118.stdout:Mon IP `172.21.15.118` is in CIDR network `172.21.0.0/20` 2024-03-12T03:15:15.031 INFO:teuthology.orchestra.run.smithi118.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-12T03:15:15.031 INFO:teuthology.orchestra.run.smithi118.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T03:15:15.032 INFO:teuthology.orchestra.run.smithi118.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-03-12T03:15:15.443 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 2024-03-12T03:15:15.444 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-03-12T03:15:15.444 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Getting image source signatures 2024-03-12T03:15:15.444 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Copying blob sha256:2c5ae85a2b9f2467483197e59b9259d5a6c90b3af6f7ed0d33be9ca6e2fedebb 2024-03-12T03:15:15.444 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-12T03:15:15.444 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Copying config sha256:fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 2024-03-12T03:15:15.444 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-12T03:15:16.398 INFO:teuthology.orchestra.run.smithi118.stdout:ceph: stdout ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable) 2024-03-12T03:15:16.399 INFO:teuthology.orchestra.run.smithi118.stdout:Ceph version: ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable) 2024-03-12T03:15:16.399 INFO:teuthology.orchestra.run.smithi118.stdout:Extracting ceph user uid/gid from container image... 2024-03-12T03:15:17.229 INFO:teuthology.orchestra.run.smithi118.stdout:stat: stdout 167 167 2024-03-12T03:15:17.230 INFO:teuthology.orchestra.run.smithi118.stdout:Creating initial keys... 2024-03-12T03:15:18.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph-authtool: stdout AQDFyO9lGxTnIBAAEsYGn5X6fBiURGS/vLkt+Q== 2024-03-12T03:15:19.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph-authtool: stdout AQDGyO9ljXFPHhAAsAkNF0WNcXRDxJFpTlaI2A== 2024-03-12T03:15:20.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph-authtool: stdout AQDHyO9l/ghOIhAA/wCMORW8FesBb+N7PVfcTA== 2024-03-12T03:15:20.167 INFO:teuthology.orchestra.run.smithi118.stdout:Creating initial monmap... 2024-03-12T03:15:21.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:15:21.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-03-12T03:15:21.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:21.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:15:21.156 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool for smithi118 [v2:172.21.15.118:3300,v1:172.21.15.118:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:15:21.156 INFO:teuthology.orchestra.run.smithi118.stdout:setting min_mon_release = pacific 2024-03-12T03:15:21.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: set fsid to 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:21.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:15:21.157 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:15:21.157 INFO:teuthology.orchestra.run.smithi118.stdout:Creating mon... 2024-03-12T03:15:22.170 INFO:teuthology.orchestra.run.smithi118.stdout:create mon.smithi118 on 2024-03-12T03:15:22.376 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-12T03:15:22.524 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T03:15:22.707 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715.target. 2024-03-12T03:15:22.707 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715.target. 2024-03-12T03:15:22.968 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118 2024-03-12T03:15:22.969 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118.service: Unit ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118.service not loaded. 2024-03-12T03:15:23.135 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715.target.wants/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118.service → /etc/systemd/system/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@.service. 2024-03-12T03:15:23.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 podman[28771]: 2024-03-12 03:15:23.280523152 +0000 UTC m=+0.017525590 image pull fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 quay.ceph.io/ceph-ci/ceph:reef 2024-03-12T03:15:23.839 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 podman[28771]: 2024-03-12 03:15:23.560375372 +0000 UTC m=+0.297377809 container create fe96ded47e84963bf23a248665052ce2078c5820a3960667980af78f5b4d77d8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-62a11b3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-12T03:15:23.839 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 podman[28771]: 2024-03-12 03:15:23.805055853 +0000 UTC m=+0.542058306 container init fe96ded47e84963bf23a248665052ce2078c5820a3960667980af78f5b4d77d8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=reef-62a11b3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:15:23.839 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 podman[28771]: 2024-03-12 03:15:23.808314911 +0000 UTC m=+0.545317350 container start fe96ded47e84963bf23a248665052ce2078c5820a3960667980af78f5b4d77d8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=reef-62a11b3, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:15:23.905 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:15:23.906 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-12T03:15:23.911 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:15:23.911 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-12T03:15:23.911 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-12T03:15:23.911 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:15:23.911 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mon to start... 2024-03-12T03:15:23.911 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mon... 2024-03-12T03:15:24.142 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 ceph-mon[28790]: mkfs 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:24.143 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 ceph-mon[28790]: mon.smithi118 is new leader, mons smithi118 in quorum (ranks 0) 2024-03-12T03:15:24.143 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 bash[28771]: fe96ded47e84963bf23a248665052ce2078c5820a3960667980af78f5b4d77d8 2024-03-12T03:15:24.143 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:23 smithi118 systemd[1]: Started Ceph mon.smithi118 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:15:24.971 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:24 smithi118 ceph-mon[28790]: mon.smithi118 is new leader, mons smithi118 in quorum (ranks 0) 2024-03-12T03:15:24.971 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:24 smithi118 ceph-mon[28790]: monmap e1: 1 mons at {smithi118=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:15:24.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:24 smithi118 ceph-mon[28790]: fsmap 2024-03-12T03:15:24.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:24 smithi118 ceph-mon[28790]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:15:24.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:24 smithi118 ceph-mon[28790]: mgrmap e1: no daemons active 2024-03-12T03:15:24.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:24 smithi118 ceph-mon[28790]: from='client.? 172.21.15.118:0/256807311' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout cluster: 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout id: 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout services: 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi118 (age 0.574115s) 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-12T03:15:24.980 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout data: 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout pgs: 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.355+0000 7f45e74b3700 1 Processor -- start 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.355+0000 7f45e74b3700 1 -- start start 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.356+0000 7f45e74b3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e0105620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.356+0000 7f45e74b3700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e0105bf0 con 0x7f45e0105200 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.356+0000 7f45e524f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e0105620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.356+0000 7f45e524f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e0105620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40092/0 (socket says 172.21.15.118:40092) 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.356+0000 7f45e524f700 1 -- 172.21.15.118:0/1109488169 learned_addr learned my addr 172.21.15.118:0/1109488169 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:24.981 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.356+0000 7f45e524f700 1 -- 172.21.15.118:0/1109488169 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0106400 con 0x7f45e0105200 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.356+0000 7f45e524f700 1 --2- 172.21.15.118:0/1109488169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e0105620 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f45d00098a0 tx=0x7f45d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c876a4882880856d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45d7fff700 1 -- 172.21.15.118:0/1109488169 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45d0004180 con 0x7f45e0105200 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45d7fff700 1 -- 172.21.15.118:0/1109488169 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d00042e0 con 0x7f45e0105200 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45d7fff700 1 -- 172.21.15.118:0/1109488169 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45d00036a0 con 0x7f45e0105200 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45e74b3700 1 -- 172.21.15.118:0/1109488169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 msgr2=0x7f45e0105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45e74b3700 1 --2- 172.21.15.118:0/1109488169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e0105620 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f45d00098a0 tx=0x7f45d0009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45e74b3700 1 -- 172.21.15.118:0/1109488169 shutdown_connections 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45e74b3700 1 --2- 172.21.15.118:0/1109488169 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e0105620 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45e74b3700 1 -- 172.21.15.118:0/1109488169 >> 172.21.15.118:0/1109488169 conn(0x7f45e01007a0 msgr2=0x7f45e0102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45e74b3700 1 -- 172.21.15.118:0/1109488169 shutdown_connections 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.357+0000 7f45e74b3700 1 -- 172.21.15.118:0/1109488169 wait complete. 2024-03-12T03:15:24.982 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e74b3700 1 Processor -- start 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e74b3700 1 -- start start 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e74b3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e00740d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e74b3700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e0074610 con 0x7f45e0105200 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e524f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e00740d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e524f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e00740d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40104/0 (socket says 172.21.15.118:40104) 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e524f700 1 -- 172.21.15.118:0/256807311 learned_addr learned my addr 172.21.15.118:0/256807311 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e524f700 1 -- 172.21.15.118:0/256807311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d0009550 con 0x7f45e0105200 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.358+0000 7f45e524f700 1 --2- 172.21.15.118:0/256807311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e00740d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f45d0004000 tx=0x7f45d0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.359+0000 7f45d67fc700 1 -- 172.21.15.118:0/256807311 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45d0003ce0 con 0x7f45e0105200 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.359+0000 7f45d67fc700 1 -- 172.21.15.118:0/256807311 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0020070 con 0x7f45e0105200 2024-03-12T03:15:24.983 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.359+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e019c580 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.359+0000 7f45d67fc700 1 -- 172.21.15.118:0/256807311 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45d0011630 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.359+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e019c9c0 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.359+0000 7f45d67fc700 1 -- 172.21.15.118:0/256807311 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f45d0022030 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.359+0000 7f45d67fc700 1 -- 172.21.15.118:0/256807311 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f45d0025d40 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.360+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e01924c0 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.362+0000 7f45d67fc700 1 -- 172.21.15.118:0/256807311 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f45d0025480 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.430+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f45e002ce60 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.431+0000 7f45d67fc700 1 -- 172.21.15.118:0/256807311 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f45d0036070 con 0x7f45e0105200 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.431+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 msgr2=0x7f45e00740d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.431+0000 7f45e74b3700 1 --2- 172.21.15.118:0/256807311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e00740d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f45d0004000 tx=0x7f45d0004750 comp rx=0 tx=0).stop 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.431+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 shutdown_connections 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.431+0000 7f45e74b3700 1 --2- 172.21.15.118:0/256807311 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45e0105200 0x7f45e00740d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.431+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 >> 172.21.15.118:0/256807311 conn(0x7f45e01007a0 msgr2=0x7f45e018f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:24.984 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.432+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 shutdown_connections 2024-03-12T03:15:24.985 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:24.432+0000 7f45e74b3700 1 -- 172.21.15.118:0/256807311 wait complete. 2024-03-12T03:15:24.985 INFO:teuthology.orchestra.run.smithi118.stdout:mon is available 2024-03-12T03:15:24.985 INFO:teuthology.orchestra.run.smithi118.stdout:Assimilating anything we can from ceph.conf... 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout fsid = 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.118:3300,v1:172.21.15.118:6789] 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:15:26.037 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.401+0000 7ff9f23ef700 1 Processor -- start 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.401+0000 7ff9f23ef700 1 -- start start 2024-03-12T03:15:26.038 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.401+0000 7ff9f23ef700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec1091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.401+0000 7ff9f23ef700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec074500 con 0x7ff9ec106df0 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9ebfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec1091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9ebfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec1091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35814/0 (socket says 172.21.15.118:35814) 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9ebfff700 1 -- 172.21.15.118:0/4271898578 learned_addr learned my addr 172.21.15.118:0/4271898578 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9ebfff700 1 -- 172.21.15.118:0/4271898578 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec109720 con 0x7ff9ec106df0 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9ebfff700 1 --2- 172.21.15.118:0/4271898578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec1091e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098a0 tx=0x7ff9dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f12684ba6b2d48f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9eaffd700 1 -- 172.21.15.118:0/4271898578 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc004180 con 0x7ff9ec106df0 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9eaffd700 1 -- 172.21.15.118:0/4271898578 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc0042e0 con 0x7ff9ec106df0 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.402+0000 7ff9eaffd700 1 -- 172.21.15.118:0/4271898578 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc0036a0 con 0x7ff9ec106df0 2024-03-12T03:15:26.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 -- 172.21.15.118:0/4271898578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 msgr2=0x7ff9ec1091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 --2- 172.21.15.118:0/4271898578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec1091e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098a0 tx=0x7ff9dc009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 -- 172.21.15.118:0/4271898578 shutdown_connections 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 --2- 172.21.15.118:0/4271898578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec1091e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 -- 172.21.15.118:0/4271898578 >> 172.21.15.118:0/4271898578 conn(0x7ff9ec1007a0 msgr2=0x7ff9ec102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 -- 172.21.15.118:0/4271898578 shutdown_connections 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 -- 172.21.15.118:0/4271898578 wait complete. 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.403+0000 7ff9f23ef700 1 Processor -- start 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9f23ef700 1 -- start start 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9f23ef700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9f23ef700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec199d50 con 0x7ff9ec106df0 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9ebfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9ebfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35828/0 (socket says 172.21.15.118:35828) 2024-03-12T03:15:26.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9ebfff700 1 -- 172.21.15.118:0/316492524 learned_addr learned my addr 172.21.15.118:0/316492524 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9ebfff700 1 -- 172.21.15.118:0/316492524 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc009550 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9ebfff700 1 --2- 172.21.15.118:0/316492524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec199810 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec073940 tx=0x7ff9dc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.404+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc003ce0 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.405+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9dc020070 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.405+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec199f50 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.405+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9dc011630 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.405+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec19a3f0 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.405+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff9dc022030 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.405+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff9dc025d00 con 0x7ff9ec106df0 2024-03-12T03:15:26.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.406+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ec193600 con 0x7ff9ec106df0 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.408+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7ff9dc025480 con 0x7ff9ec106df0 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.476+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff9ec02cf10 con 0x7ff9ec106df0 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.479+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 7 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ff9dc011c20 con 0x7ff9ec106df0 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.479+0000 7ff9e97fa700 1 -- 172.21.15.118:0/316492524 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7ff9dc01c500 con 0x7ff9ec106df0 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.480+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 msgr2=0x7ff9ec199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.480+0000 7ff9f23ef700 1 --2- 172.21.15.118:0/316492524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec199810 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec073940 tx=0x7ff9dc004750 comp rx=0 tx=0).stop 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.480+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 shutdown_connections 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.480+0000 7ff9f23ef700 1 --2- 172.21.15.118:0/316492524 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec106df0 0x7ff9ec199810 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.480+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 >> 172.21.15.118:0/316492524 conn(0x7ff9ec1007a0 msgr2=0x7ff9ec100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.481+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 shutdown_connections 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:25.481+0000 7ff9f23ef700 1 -- 172.21.15.118:0/316492524 wait complete. 2024-03-12T03:15:26.042 INFO:teuthology.orchestra.run.smithi118.stdout:Generating new minimal ceph.conf... 2024-03-12T03:15:26.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:25 smithi118 ceph-mon[28790]: from='client.? 172.21.15.118:0/316492524' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:15:26.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:25 smithi118 ceph-mon[28790]: from='client.? 172.21.15.118:0/316492524' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T03:15:27.135 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:26 smithi118 ceph-mon[28790]: from='client.? 172.21.15.118:0/3402588323' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:15:27.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.479+0000 7f485e89f700 1 Processor -- start 2024-03-12T03:15:27.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.479+0000 7f485e89f700 1 -- start start 2024-03-12T03:15:27.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.479+0000 7f485e89f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f4858106750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:27.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.479+0000 7f485e89f700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4858106d20 con 0x7f4858106330 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.479+0000 7f4857fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f4858106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.479+0000 7f4857fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f4858106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35836/0 (socket says 172.21.15.118:35836) 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.479+0000 7f4857fff700 1 -- 172.21.15.118:0/3551434540 learned_addr learned my addr 172.21.15.118:0/3551434540 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f4857fff700 1 -- 172.21.15.118:0/3551434540 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4858107530 con 0x7f4858106330 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f4857fff700 1 --2- 172.21.15.118:0/3551434540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f4858106750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48480098a0 tx=0x7f4848009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9703f3b4ad1293a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f4856ffd700 1 -- 172.21.15.118:0/3551434540 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4848004180 con 0x7f4858106330 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f4856ffd700 1 -- 172.21.15.118:0/3551434540 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f48480042e0 con 0x7f4858106330 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f4856ffd700 1 -- 172.21.15.118:0/3551434540 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4848011a70 con 0x7f4858106330 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f485e89f700 1 -- 172.21.15.118:0/3551434540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 msgr2=0x7f4858106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f485e89f700 1 --2- 172.21.15.118:0/3551434540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f4858106750 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f48480098a0 tx=0x7f4848009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f485e89f700 1 -- 172.21.15.118:0/3551434540 shutdown_connections 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f485e89f700 1 --2- 172.21.15.118:0/3551434540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f4858106750 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:27.142 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f485e89f700 1 -- 172.21.15.118:0/3551434540 >> 172.21.15.118:0/3551434540 conn(0x7f48581018b0 msgr2=0x7f4858103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f485e89f700 1 -- 172.21.15.118:0/3551434540 shutdown_connections 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.480+0000 7f485e89f700 1 -- 172.21.15.118:0/3551434540 wait complete. 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.481+0000 7f485e89f700 1 Processor -- start 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.481+0000 7f485e89f700 1 -- start start 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.481+0000 7f485e89f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f485807ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.481+0000 7f485e89f700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485807bfa0 con 0x7f4858106330 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.481+0000 7f4857fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f485807ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.481+0000 7f4857fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f485807ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35852/0 (socket says 172.21.15.118:35852) 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.481+0000 7f4857fff700 1 -- 172.21.15.118:0/3402588323 learned_addr learned my addr 172.21.15.118:0/3402588323 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f4857fff700 1 -- 172.21.15.118:0/3402588323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848009550 con 0x7f4858106330 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f4857fff700 1 --2- 172.21.15.118:0/3402588323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f485807ba60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4848011710 tx=0x7f48480047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f48557fa700 1 -- 172.21.15.118:0/3402588323 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4848003b10 con 0x7f4858106330 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f48557fa700 1 -- 172.21.15.118:0/3402588323 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f484801b8c0 con 0x7f4858106330 2024-03-12T03:15:27.143 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485807a1e0 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f48557fa700 1 -- 172.21.15.118:0/3402588323 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4848019710 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485807a680 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f48557fa700 1 -- 172.21.15.118:0/3402588323 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4848019be0 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.482+0000 7f48557fa700 1 -- 172.21.15.118:0/3402588323 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f484801a710 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.483+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48580623e0 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.485+0000 7f48557fa700 1 -- 172.21.15.118:0/3402588323 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f484801ba30 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.551+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f485810a760 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.552+0000 7f48557fa700 1 -- 172.21.15.118:0/3402588323 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f4848030090 con 0x7f4858106330 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.552+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 msgr2=0x7f485807ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.552+0000 7f485e89f700 1 --2- 172.21.15.118:0/3402588323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f485807ba60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4848011710 tx=0x7f48480047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.552+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 shutdown_connections 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.552+0000 7f485e89f700 1 --2- 172.21.15.118:0/3402588323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4858106330 0x7f485807ba60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.552+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 >> 172.21.15.118:0/3402588323 conn(0x7f48581018b0 msgr2=0x7f4858102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.553+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 shutdown_connections 2024-03-12T03:15:27.144 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:26.553+0000 7f485e89f700 1 -- 172.21.15.118:0/3402588323 wait complete. 2024-03-12T03:15:27.145 INFO:teuthology.orchestra.run.smithi118.stdout:Restarting the monitor... 2024-03-12T03:15:27.422 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:27 smithi118 systemd[1]: Stopping Ceph mon.smithi118 for 9d8e7844-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:15:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[28785]: 2024-03-12T03:15:27.419+0000 7f3c38745700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi118 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:15:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[28785]: 2024-03-12T03:15:27.419+0000 7f3c38745700 -1 mon.smithi118@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T03:15:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:27 smithi118 podman[29035]: 2024-03-12 03:15:27.624248389 +0000 UTC m=+0.331014395 container died fe96ded47e84963bf23a248665052ce2078c5820a3960667980af78f5b4d77d8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=reef-62a11b3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T03:15:28.122 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:27 smithi118 podman[29035]: 2024-03-12 03:15:27.936301131 +0000 UTC m=+0.643067136 container cleanup fe96ded47e84963bf23a248665052ce2078c5820a3960667980af78f5b4d77d8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=reef-62a11b3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-12T03:15:28.122 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:27 smithi118 bash[29035]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118 2024-03-12T03:15:28.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 podman[29049]: 2024-03-12 03:15:28.12066663 +0000 UTC m=+0.494497582 container remove fe96ded47e84963bf23a248665052ce2078c5820a3960667980af78f5b4d77d8 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=reef-62a11b3, org.label-schema.build-date=20240307) 2024-03-12T03:15:28.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118.service: Deactivated successfully. 2024-03-12T03:15:28.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 systemd[1]: Stopped Ceph mon.smithi118 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:15:28.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 systemd[1]: Starting Ceph mon.smithi118 for 9d8e7844-e01e-11ee-95c8-87774f69a715... 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 podman[29141]: 2024-03-12 03:15:28.464567692 +0000 UTC m=+0.018500706 image pull fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 quay.ceph.io/ceph-ci/ceph:reef 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 podman[29141]: 2024-03-12 03:15:28.563208671 +0000 UTC m=+0.117141714 container create a26ddd75d886f21bf923031e4897c2faa28d79e845d24b17902134c8170bea36 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=reef-62a11b3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 podman[29141]: 2024-03-12 03:15:28.757111853 +0000 UTC m=+0.311044857 container init a26ddd75d886f21bf923031e4897c2faa28d79e845d24b17902134c8170bea36 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=reef-62a11b3, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 podman[29141]: 2024-03-12 03:15:28.760181909 +0000 UTC m=+0.314114917 container start a26ddd75d886f21bf923031e4897c2faa28d79e845d24b17902134c8170bea36 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-62a11b3, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable), process ceph-mon, pid 2 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: pidfile_write: ignore empty --pid-file 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: load: jerasure load: lrc 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Git sha 0 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Compile date 2024-03-11 17:58:16 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: DB SUMMARY 2024-03-12T03:15:28.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: DB Session ID: 7TANLUOWG573F5NFGC3X 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi118/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi118/store.db: 000009.log size: 97610 ; 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:15:28.862 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:15:28.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:15:28.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.env: 0x55c66f309720 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.info_log: 0x55c671267380 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.statistics: (nil) 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.use_fsync: 0 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:15:28.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.db_log_dir: 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.wal_dir: 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.write_buffer_manager: 0x55c6704e0140 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:15:28.865 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.unordered_write: 0 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.row_cache: None 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.wal_filter: None 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:15:28.866 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.wal_compression: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:15:28.867 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_open_files: -1 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:15:28.868 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Compression algorithms supported: 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kZSTD supported: 0 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:15:28.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi118/store.db/MANIFEST-000010 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.merge_operator: 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_filter: None 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c6712674a0) 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: index_type: 0 2024-03-12T03:15:28.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: data_block_index_type: 0 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: index_shortening: 1 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: checksum: 4 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: no_block_cache: 0 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_cache: 0x55c6705811f0 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_cache_options: 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: capacity : 536870912 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: num_shard_bits : 4 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: strict_capacity_limit : 0 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_cache_compressed: (nil) 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: persistent_cache: (nil) 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_size: 4096 2024-03-12T03:15:28.871 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_size_deviation: 10 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_restart_interval: 16 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: index_block_restart_interval: 1 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: metadata_block_size: 4096 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: partition_filters: 0 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: use_delta_encoding: 1 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: filter_policy: bloomfilter 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: whole_key_filtering: 1 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: verify_compression: 0 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: format_version: 5 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: enable_index_compression: 1 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: block_align: 0 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: max_auto_readahead_size: 262144 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: prepopulate_block_cache: 0 2024-03-12T03:15:28.872 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression: NoCompression 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.num_levels: 7 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:15:28.873 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:15:28.874 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:15:28.875 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:15:28.876 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.ttl: 2592000 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:15:28.877 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi118/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 89f674a6-0362-4069-8b51-25a80a18fede 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213328788720, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-12T03:15:28.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213328789717, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92960, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 90977, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83692, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710213328, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "89f674a6-0362-4069-8b51-25a80a18fede", "db_session_id": "7TANLUOWG573F5NFGC3X", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213328789797, "job": 1, "event": "recovery_finished"} 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi118/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c67061e000 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: DB pointer 0x55c67060c000 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: ** DB Stats ** 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:15:28.879 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: ** Compaction Stats [default] ** 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: L0 2/0 92.65 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Sum 2/0 92.65 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 139.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: ** Compaction Stats [default] ** 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 139.4 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:15:28.880 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Cumulative compaction: 0.00 GB write, 30.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Interval compaction: 0.00 GB write, 30.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Block cache BinnedLRUCache@0x55c6705811f0#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:15:28.881 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: starting mon.smithi118 rank 0 at public addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] at bind addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi118 fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???) e1 preinit fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???).mds e1 new map 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???).mds e1 print_map 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: e1 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: legacy client fscid: -1 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout: No filesystems configured 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:15:28.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T03:15:28.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mon.smithi118 is new leader, mons smithi118 in quorum (ranks 0) 2024-03-12T03:15:28.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: monmap e1: 1 mons at {smithi118=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:15:28.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: fsmap 2024-03-12T03:15:28.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:15:28.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 ceph-mon[29160]: mgrmap e1: no daemons active 2024-03-12T03:15:28.883 INFO:teuthology.orchestra.run.smithi118.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-12T03:15:29.119 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 bash[29141]: a26ddd75d886f21bf923031e4897c2faa28d79e845d24b17902134c8170bea36 2024-03-12T03:15:29.119 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:28 smithi118 systemd[1]: Started Ceph mon.smithi118 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.304+0000 7f7145075700 1 Processor -- start 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.304+0000 7f7145075700 1 -- start start 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.304+0000 7f7145075700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140105650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.304+0000 7f7145075700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7140105c20 con 0x7f7140105230 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.304+0000 7f713ed9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140105650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.304+0000 7f713ed9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140105650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35858/0 (socket says 172.21.15.118:35858) 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.304+0000 7f713ed9d700 1 -- 172.21.15.118:0/1045404473 learned_addr learned my addr 172.21.15.118:0/1045404473 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.305+0000 7f713ed9d700 1 -- 172.21.15.118:0/1045404473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7140106430 con 0x7f7140105230 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.305+0000 7f713ed9d700 1 --2- 172.21.15.118:0/1045404473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140105650 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f71300098a0 tx=0x7f7130009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb7b87a8485885 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:29.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.305+0000 7f713dd9b700 1 -- 172.21.15.118:0/1045404473 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130004180 con 0x7f7140105230 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.305+0000 7f713dd9b700 1 -- 172.21.15.118:0/1045404473 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f71300042e0 con 0x7f7140105230 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.305+0000 7f713dd9b700 1 -- 172.21.15.118:0/1045404473 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130011a70 con 0x7f7140105230 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.305+0000 7f7145075700 1 -- 172.21.15.118:0/1045404473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 msgr2=0x7f7140105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.305+0000 7f7145075700 1 --2- 172.21.15.118:0/1045404473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140105650 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f71300098a0 tx=0x7f7130009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.306+0000 7f7145075700 1 -- 172.21.15.118:0/1045404473 shutdown_connections 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.306+0000 7f7145075700 1 --2- 172.21.15.118:0/1045404473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140105650 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.306+0000 7f7145075700 1 -- 172.21.15.118:0/1045404473 >> 172.21.15.118:0/1045404473 conn(0x7f7140100830 msgr2=0x7f7140102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.306+0000 7f7145075700 1 -- 172.21.15.118:0/1045404473 shutdown_connections 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.306+0000 7f7145075700 1 -- 172.21.15.118:0/1045404473 wait complete. 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.306+0000 7f7145075700 1 Processor -- start 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.306+0000 7f7145075700 1 -- start start 2024-03-12T03:15:29.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f7145075700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140079960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f7145075700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7140079ea0 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f713ed9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140079960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f713ed9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140079960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35862/0 (socket says 172.21.15.118:35862) 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f713ed9d700 1 -- 172.21.15.118:0/2435425283 learned_addr learned my addr 172.21.15.118:0/2435425283 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f713ed9d700 1 -- 172.21.15.118:0/2435425283 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7130009550 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f713ed9d700 1 --2- 172.21.15.118:0/2435425283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140079960 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7130011710 tx=0x7f71300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f7127fff700 1 -- 172.21.15.118:0/2435425283 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130003b10 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f7127fff700 1 -- 172.21.15.118:0/2435425283 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f713001b8c0 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714019cdb0 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f7127fff700 1 -- 172.21.15.118:0/2435425283 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7130019710 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.307+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714019d250 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.308+0000 7f7127fff700 1 -- 172.21.15.118:0/2435425283 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7130019be0 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.308+0000 7f7127fff700 1 -- 172.21.15.118:0/2435425283 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f713001a710 con 0x7f7140105230 2024-03-12T03:15:29.883 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.308+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7140109660 con 0x7f7140105230 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.311+0000 7f7127fff700 1 -- 172.21.15.118:0/2435425283 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f713001ba30 con 0x7f7140105230 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.376+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f714019d8b0 con 0x7f7140105230 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.378+0000 7f7127fff700 1 -- 172.21.15.118:0/2435425283 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f71300300c0 con 0x7f7140105230 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.379+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 msgr2=0x7f7140079960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.379+0000 7f7145075700 1 --2- 172.21.15.118:0/2435425283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140079960 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7130011710 tx=0x7f71300047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.379+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 shutdown_connections 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.379+0000 7f7145075700 1 --2- 172.21.15.118:0/2435425283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7140105230 0x7f7140079960 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.379+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 >> 172.21.15.118:0/2435425283 conn(0x7f7140100830 msgr2=0x7f71401013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.379+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 shutdown_connections 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:29.379+0000 7f7145075700 1 -- 172.21.15.118:0/2435425283 wait complete. 2024-03-12T03:15:29.884 INFO:teuthology.orchestra.run.smithi118.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-12T03:15:29.899 INFO:teuthology.orchestra.run.smithi118.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:15:29.899 INFO:teuthology.orchestra.run.smithi118.stdout:Creating mgr... 2024-03-12T03:15:29.899 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-12T03:15:29.900 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-12T03:15:29.900 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying port 0.0.0.0:8443 ... 2024-03-12T03:15:30.218 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mgr.smithi118.zzzuvl 2024-03-12T03:15:30.218 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mgr.smithi118.zzzuvl.service: Unit ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mgr.smithi118.zzzuvl.service not loaded. 2024-03-12T03:15:30.377 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715.target.wants/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mgr.smithi118.zzzuvl.service → /etc/systemd/system/ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@.service. 2024-03-12T03:15:30.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:30 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2435425283' entity='client.admin' 2024-03-12T03:15:30.933 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:15:30.933 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-12T03:15:30.940 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:15:30.940 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-12T03:15:30.940 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-12T03:15:30.940 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to enable service . firewalld.service is not available 2024-03-12T03:15:30.948 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:15:30.948 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-12T03:15:30.955 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:15:30.955 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-12T03:15:30.955 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-12T03:15:30.955 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-03-12T03:15:30.955 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr to start... 2024-03-12T03:15:30.955 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr... 2024-03-12T03:15:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:31 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1382461690' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:15:32.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "9d8e7844-e01e-11ee-95c8-87774f69a715", 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:15:32.091 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "smithi118" 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:15:32.092 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:15:32.100 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:32.101 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:15:23.859067+0000", 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.362+0000 7f866e037700 1 Processor -- start 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.362+0000 7f866e037700 1 -- start start 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.362+0000 7f866e037700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f86681091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.362+0000 7f866e037700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8668079db0 con 0x7f8668106df0 2024-03-12T03:15:32.102 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86677fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f86681091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:32.105 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86677fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f86681091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35884/0 (socket says 172.21.15.118:35884) 2024-03-12T03:15:32.105 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86677fe700 1 -- 172.21.15.118:0/1854030476 learned_addr learned my addr 172.21.15.118:0/1854030476 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:32.105 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86677fe700 1 -- 172.21.15.118:0/1854030476 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8668109720 con 0x7f8668106df0 2024-03-12T03:15:32.105 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86677fe700 1 --2- 172.21.15.118:0/1854030476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f86681091e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f865c0098a0 tx=0x7f865c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c344373859fcf6f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:32.105 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86667fc700 1 -- 172.21.15.118:0/1854030476 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f865c004180 con 0x7f8668106df0 2024-03-12T03:15:32.105 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86667fc700 1 -- 172.21.15.118:0/1854030476 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f865c0042e0 con 0x7f8668106df0 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.363+0000 7f86667fc700 1 -- 172.21.15.118:0/1854030476 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f865c011a70 con 0x7f8668106df0 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.364+0000 7f866e037700 1 -- 172.21.15.118:0/1854030476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 msgr2=0x7f86681091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.364+0000 7f866e037700 1 --2- 172.21.15.118:0/1854030476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f86681091e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f865c0098a0 tx=0x7f865c009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.364+0000 7f866e037700 1 -- 172.21.15.118:0/1854030476 shutdown_connections 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.364+0000 7f866e037700 1 --2- 172.21.15.118:0/1854030476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f86681091e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.364+0000 7f866e037700 1 -- 172.21.15.118:0/1854030476 >> 172.21.15.118:0/1854030476 conn(0x7f8668100840 msgr2=0x7f8668102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.364+0000 7f866e037700 1 -- 172.21.15.118:0/1854030476 shutdown_connections 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.364+0000 7f866e037700 1 -- 172.21.15.118:0/1854030476 wait complete. 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f866e037700 1 Processor -- start 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f866e037700 1 -- start start 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f866e037700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f866819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f866e037700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866819a5a0 con 0x7f8668106df0 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f86677fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f866819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f86677fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f866819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35892/0 (socket says 172.21.15.118:35892) 2024-03-12T03:15:32.106 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f86677fe700 1 -- 172.21.15.118:0/1382461690 learned_addr learned my addr 172.21.15.118:0/1382461690 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.365+0000 7f86677fe700 1 -- 172.21.15.118:0/1382461690 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c009550 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f86677fe700 1 --2- 172.21.15.118:0/1382461690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f866819a060 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f865c011710 tx=0x7f865c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f8664ff9700 1 -- 172.21.15.118:0/1382461690 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f865c003b10 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866819a7a0 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f8664ff9700 1 -- 172.21.15.118:0/1382461690 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f865c01b8c0 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f8664ff9700 1 -- 172.21.15.118:0/1382461690 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f865c0197e0 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866819ac40 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f8664ff9700 1 -- 172.21.15.118:0/1382461690 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f865c019cd0 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.366+0000 7f8664ff9700 1 -- 172.21.15.118:0/1382461690 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f865c01a7e0 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.367+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866810a470 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.369+0000 7f8664ff9700 1 -- 172.21.15.118:0/1382461690 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f865c029060 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.454+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f866810a630 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.455+0000 7f8664ff9700 1 -- 172.21.15.118:0/1382461690 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f865c01f070 con 0x7f8668106df0 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.456+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 msgr2=0x7f866819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:32.107 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.456+0000 7f866e037700 1 --2- 172.21.15.118:0/1382461690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f866819a060 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f865c011710 tx=0x7f865c0047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:32.108 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.456+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 shutdown_connections 2024-03-12T03:15:32.108 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.456+0000 7f866e037700 1 --2- 172.21.15.118:0/1382461690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8668106df0 0x7f866819a060 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:32.108 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.456+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 >> 172.21.15.118:0/1382461690 conn(0x7f8668100840 msgr2=0x7f86681013e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:32.108 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.456+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 shutdown_connections 2024-03-12T03:15:32.108 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:31.456+0000 7f866e037700 1 -- 172.21.15.118:0/1382461690 wait complete. 2024-03-12T03:15:32.108 INFO:teuthology.orchestra.run.smithi118.stdout:mgr not available, waiting (1/15)... 2024-03-12T03:15:35.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:34 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/576029589' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "9d8e7844-e01e-11ee-95c8-87774f69a715", 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:15:35.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "smithi118" 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:15:35.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:15:35.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:15:35.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:15:23.859067+0000", 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:15:35.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.537+0000 7f1800d6b700 1 Processor -- start 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.537+0000 7f1800d6b700 1 -- start start 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.537+0000 7f1800d6b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc104d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.537+0000 7f1800d6b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc105360 con 0x7f17fc104970 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.537+0000 7f17fa59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.537+0000 7f17fa59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc104d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40352/0 (socket says 172.21.15.118:40352) 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.537+0000 7f17fa59c700 1 -- 172.21.15.118:0/4039591893 learned_addr learned my addr 172.21.15.118:0/4039591893 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.538+0000 7f17fa59c700 1 -- 172.21.15.118:0/4039591893 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc105b70 con 0x7f17fc104970 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.538+0000 7f17fa59c700 1 --2- 172.21.15.118:0/4039591893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc104d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00aa40 tx=0x7f17ec00ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6f9f99c2f25a79 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.538+0000 7f17f959a700 1 -- 172.21.15.118:0/4039591893 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec004180 con 0x7f17fc104970 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.538+0000 7f17f959a700 1 -- 172.21.15.118:0/4039591893 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f17ec00cd70 con 0x7f17fc104970 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.538+0000 7f17f959a700 1 -- 172.21.15.118:0/4039591893 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec003cf0 con 0x7f17fc104970 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.539+0000 7f1800d6b700 1 -- 172.21.15.118:0/4039591893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 msgr2=0x7f17fc104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:35.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.539+0000 7f1800d6b700 1 --2- 172.21.15.118:0/4039591893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc104d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00aa40 tx=0x7f17ec00ad50 comp rx=0 tx=0).stop 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.539+0000 7f1800d6b700 1 -- 172.21.15.118:0/4039591893 shutdown_connections 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.539+0000 7f1800d6b700 1 --2- 172.21.15.118:0/4039591893 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc104d90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.539+0000 7f1800d6b700 1 -- 172.21.15.118:0/4039591893 >> 172.21.15.118:0/4039591893 conn(0x7f17fc0fff30 msgr2=0x7f17fc102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.539+0000 7f1800d6b700 1 -- 172.21.15.118:0/4039591893 shutdown_connections 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.539+0000 7f1800d6b700 1 -- 172.21.15.118:0/4039591893 wait complete. 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f1800d6b700 1 Processor -- start 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f1800d6b700 1 -- start start 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f1800d6b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f1800d6b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc1983d0 con 0x7f17fc104970 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f17fa59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f17fa59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40364/0 (socket says 172.21.15.118:40364) 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f17fa59c700 1 -- 172.21.15.118:0/576029589 learned_addr learned my addr 172.21.15.118:0/576029589 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.540+0000 7f17fa59c700 1 -- 172.21.15.118:0/576029589 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec00a6f0 con 0x7f17fc104970 2024-03-12T03:15:35.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f17fa59c700 1 --2- 172.21.15.118:0/576029589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc197e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17ec006d50 tx=0x7f17ec004240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f17eb7fe700 1 -- 172.21.15.118:0/576029589 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec01e070 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f17eb7fe700 1 -- 172.21.15.118:0/576029589 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f17ec004490 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17fc1985d0 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f17eb7fe700 1 -- 172.21.15.118:0/576029589 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17ec022720 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17fc198a70 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f17eb7fe700 1 -- 172.21.15.118:0/576029589 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f17ec022990 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.541+0000 7f17eb7fe700 1 -- 172.21.15.118:0/576029589 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f17ec02cc90 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.542+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17fc191b90 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.544+0000 7f17eb7fe700 1 -- 172.21.15.118:0/576029589 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f17ec027070 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.614+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f17fc02ccb0 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.614+0000 7f17eb7fe700 1 -- 172.21.15.118:0/576029589 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f17ec00b040 con 0x7f17fc104970 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.615+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 msgr2=0x7f17fc197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.615+0000 7f1800d6b700 1 --2- 172.21.15.118:0/576029589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc197e90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17ec006d50 tx=0x7f17ec004240 comp rx=0 tx=0).stop 2024-03-12T03:15:35.204 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.615+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 shutdown_connections 2024-03-12T03:15:35.205 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.615+0000 7f1800d6b700 1 --2- 172.21.15.118:0/576029589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f17fc104970 0x7f17fc197e90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:35.205 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.615+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 >> 172.21.15.118:0/576029589 conn(0x7f17fc0fff30 msgr2=0x7f17fc18ecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:35.205 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.615+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 shutdown_connections 2024-03-12T03:15:35.205 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:34.615+0000 7f1800d6b700 1 -- 172.21.15.118:0/576029589 wait complete. 2024-03-12T03:15:35.205 INFO:teuthology.orchestra.run.smithi118.stdout:mgr not available, waiting (2/15)... 2024-03-12T03:15:37.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: Activating manager daemon smithi118.zzzuvl 2024-03-12T03:15:37.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: mgrmap e2: smithi118.zzzuvl(active, starting, since 0.00317777s) 2024-03-12T03:15:37.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr metadata", "who": "smithi118.zzzuvl", "id": "smithi118.zzzuvl"}]: dispatch 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: Manager daemon smithi118.zzzuvl is now available 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/trash_purge_schedule"}]: dispatch 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='mgr.14100 172.21.15.118:0/3925590160' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:15:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:37 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1717997065' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:15:38.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "9d8e7844-e01e-11ee-95c8-87774f69a715", 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "smithi118" 2024-03-12T03:15:38.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:15:38.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:15:38.206 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:15:38.206 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:15:38.206 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:15:38.206 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:38.207 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:15:23.859067+0000", 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:38.208 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.556+0000 7fa1b2c57700 1 Processor -- start 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.556+0000 7fa1b2c57700 1 -- start start 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1b2c57700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1b2c57700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac079d90 con 0x7fa1ac106de0 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1b09f3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1b09f3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40366/0 (socket says 172.21.15.118:40366) 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1b09f3700 1 -- 172.21.15.118:0/921454855 learned_addr learned my addr 172.21.15.118:0/921454855 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1b09f3700 1 -- 172.21.15.118:0/921454855 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac109710 con 0x7fa1ac106de0 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1b09f3700 1 --2- 172.21.15.118:0/921454855 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac1091d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00098a0 tx=0x7fa1a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea6049030555f7d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1ab7fe700 1 -- 172.21.15.118:0/921454855 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a0004180 con 0x7fa1ac106de0 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1ab7fe700 1 -- 172.21.15.118:0/921454855 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa1a00042e0 con 0x7fa1ac106de0 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.557+0000 7fa1ab7fe700 1 -- 172.21.15.118:0/921454855 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a0011a70 con 0x7fa1ac106de0 2024-03-12T03:15:38.209 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.558+0000 7fa1b2c57700 1 -- 172.21.15.118:0/921454855 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 msgr2=0x7fa1ac1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.558+0000 7fa1b2c57700 1 --2- 172.21.15.118:0/921454855 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac1091d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa1a00098a0 tx=0x7fa1a0009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.558+0000 7fa1b2c57700 1 -- 172.21.15.118:0/921454855 shutdown_connections 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.558+0000 7fa1b2c57700 1 --2- 172.21.15.118:0/921454855 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac1091d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.558+0000 7fa1b2c57700 1 -- 172.21.15.118:0/921454855 >> 172.21.15.118:0/921454855 conn(0x7fa1ac1007b0 msgr2=0x7fa1ac102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.558+0000 7fa1b2c57700 1 -- 172.21.15.118:0/921454855 shutdown_connections 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.558+0000 7fa1b2c57700 1 -- 172.21.15.118:0/921454855 wait complete. 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b2c57700 1 Processor -- start 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b2c57700 1 -- start start 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b2c57700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b2c57700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac19a580 con 0x7fa1ac106de0 2024-03-12T03:15:38.210 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b09f3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b09f3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac19a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40374/0 (socket says 172.21.15.118:40374) 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b09f3700 1 -- 172.21.15.118:0/1717997065 learned_addr learned my addr 172.21.15.118:0/1717997065 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b09f3700 1 -- 172.21.15.118:0/1717997065 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0009550 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.559+0000 7fa1b09f3700 1 --2- 172.21.15.118:0/1717997065 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac19a040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0011a20 tx=0x7fa1a00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.560+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a0003b10 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.560+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fa1a001b8c0 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.560+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac19a780 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.560+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a00237b0 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.560+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac19ac20 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.560+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa1a0023ce0 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.560+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa1a0019710 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.561+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac10a430 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.563+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fa1a001ba30 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.596+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 2) v1 ==== 45145+0+0 (secure 0 0 0) 0x7fa1a001bbd0 con 0x7fa1ac106de0 2024-03-12T03:15:38.211 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.639+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa1ac10a620 con 0x7fa1ac106de0 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.639+0000 7fa1a9ffb700 1 -- 172.21.15.118:0/1717997065 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fa1a002c3f0 con 0x7fa1ac106de0 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.640+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 msgr2=0x7fa1ac19a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.640+0000 7fa1b2c57700 1 --2- 172.21.15.118:0/1717997065 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac19a040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0011a20 tx=0x7fa1a00047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.640+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 shutdown_connections 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.640+0000 7fa1b2c57700 1 --2- 172.21.15.118:0/1717997065 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1ac106de0 0x7fa1ac19a040 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.640+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 >> 172.21.15.118:0/1717997065 conn(0x7fa1ac1007b0 msgr2=0x7fa1ac101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.640+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 shutdown_connections 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:37.640+0000 7fa1b2c57700 1 -- 172.21.15.118:0/1717997065 wait complete. 2024-03-12T03:15:38.212 INFO:teuthology.orchestra.run.smithi118.stdout:mgr not available, waiting (3/15)... 2024-03-12T03:15:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:39 smithi118 ceph-mon[29160]: mgrmap e3: smithi118.zzzuvl(active, since 1.00717s) 2024-03-12T03:15:41.986 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:41 smithi118 ceph-mon[29160]: mgrmap e4: smithi118.zzzuvl(active, since 3s) 2024-03-12T03:15:45.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:44 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1590916429' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:15:48.297 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:48.297 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "9d8e7844-e01e-11ee-95c8-87774f69a715", 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "smithi118" 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:48.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-12T03:15:48.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-12T03:15:48.309 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-12T03:15:48.310 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-12T03:15:23.859067+0000", 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64add6e700 1 Processor -- start 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64add6e700 1 -- start start 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64add6e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a8106750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:48.311 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64add6e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8106d20 con 0x7f64a8106330 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64a77fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a8106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64a77fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a8106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40462/0 (socket says 172.21.15.118:40462) 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64a77fe700 1 -- 172.21.15.118:0/3413464845 learned_addr learned my addr 172.21.15.118:0/3413464845 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.108+0000 7f64a77fe700 1 -- 172.21.15.118:0/3413464845 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a8107530 con 0x7f64a8106330 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64a77fe700 1 --2- 172.21.15.118:0/3413464845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a8106750 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f64980098a0 tx=0x7f6498009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688fedea605b78c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64a67fc700 1 -- 172.21.15.118:0/3413464845 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6498004180 con 0x7f64a8106330 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64a67fc700 1 -- 172.21.15.118:0/3413464845 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f64980042e0 con 0x7f64a8106330 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64a67fc700 1 -- 172.21.15.118:0/3413464845 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6498011a70 con 0x7f64a8106330 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64add6e700 1 -- 172.21.15.118:0/3413464845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 msgr2=0x7f64a8106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64add6e700 1 --2- 172.21.15.118:0/3413464845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a8106750 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f64980098a0 tx=0x7f6498009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64add6e700 1 -- 172.21.15.118:0/3413464845 shutdown_connections 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64add6e700 1 --2- 172.21.15.118:0/3413464845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a8106750 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64add6e700 1 -- 172.21.15.118:0/3413464845 >> 172.21.15.118:0/3413464845 conn(0x7f64a81018b0 msgr2=0x7f64a8103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:48.312 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64add6e700 1 -- 172.21.15.118:0/3413464845 shutdown_connections 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.109+0000 7f64add6e700 1 -- 172.21.15.118:0/3413464845 wait complete. 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64add6e700 1 Processor -- start 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64add6e700 1 -- start start 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64add6e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64add6e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a819a5e0 con 0x7f64a8106330 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64a77fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64a77fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a819a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40470/0 (socket says 172.21.15.118:40470) 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64a77fe700 1 -- 172.21.15.118:0/1590916429 learned_addr learned my addr 172.21.15.118:0/1590916429 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.110+0000 7f64a77fe700 1 -- 172.21.15.118:0/1590916429 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498009550 con 0x7f64a8106330 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.111+0000 7f64a77fe700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a819a0a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6498011710 tx=0x7f64980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:48.313 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.111+0000 7f64a4ff9700 1 -- 172.21.15.118:0/1590916429 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6498003b10 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.111+0000 7f64a4ff9700 1 -- 172.21.15.118:0/1590916429 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f649801b8c0 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.111+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a819a7e0 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.111+0000 7f64a4ff9700 1 -- 172.21.15.118:0/1590916429 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6498019710 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.111+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a819ac80 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.112+0000 7f64a4ff9700 1 -- 172.21.15.118:0/1590916429 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 45378+0+0 (secure 0 0 0) 0x7f6498019910 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.112+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a810a760 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.112+0000 7f64a4ff9700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f6490038460 0x7f649003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.112+0000 7f64a4ff9700 1 -- 172.21.15.118:0/1590916429 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f649804c7a0 con 0x7f64a8106330 2024-03-12T03:15:48.314 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.112+0000 7f64a6ffd700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f6490038460 0x7f649003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.112+0000 7f64a6ffd700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f6490038460 0x7f649003a920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f649c006fd0 tx=0x7f649c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.117+0000 7f64a4ff9700 1 -- 172.21.15.118:0/1590916429 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6498018b80 con 0x7f64a8106330 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.389+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f64a819d930 con 0x7f64a8106330 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.390+0000 7f64a4ff9700 1 -- 172.21.15.118:0/1590916429 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f6498018350 con 0x7f64a8106330 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f6490038460 msgr2=0x7f649003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f6490038460 0x7f649003a920 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f649c006fd0 tx=0x7f649c006e40 comp rx=0 tx=0).stop 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 msgr2=0x7f64a819a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a819a0a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6498011710 tx=0x7f64980047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 shutdown_connections 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f6490038460 0x7f649003a920 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 --2- 172.21.15.118:0/1590916429 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64a8106330 0x7f64a819a0a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 >> 172.21.15.118:0/1590916429 conn(0x7f64a81018b0 msgr2=0x7f64a8102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 shutdown_connections 2024-03-12T03:15:48.315 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:44.392+0000 7f64add6e700 1 -- 172.21.15.118:0/1590916429 wait complete. 2024-03-12T03:15:48.316 INFO:teuthology.orchestra.run.smithi118.stdout:mgr is available 2024-03-12T03:15:49.608 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:49.608 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [global] 2024-03-12T03:15:49.608 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout fsid = 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:15:49.608 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-12T03:15:49.608 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [osd] 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-12T03:15:49.609 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.762+0000 7f810488c700 1 Processor -- start 2024-03-12T03:15:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.762+0000 7f810488c700 1 -- start start 2024-03-12T03:15:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.763+0000 7f810488c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f8100106750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.763+0000 7f810488c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100106d20 con 0x7f8100106330 2024-03-12T03:15:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.763+0000 7f80fe59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f8100106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.763+0000 7f80fe59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f8100106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37392/0 (socket says 172.21.15.118:37392) 2024-03-12T03:15:49.610 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.763+0000 7f80fe59c700 1 -- 172.21.15.118:0/1451592334 learned_addr learned my addr 172.21.15.118:0/1451592334 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:49.611 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.763+0000 7f80fe59c700 1 -- 172.21.15.118:0/1451592334 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100107530 con 0x7f8100106330 2024-03-12T03:15:49.611 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f80fe59c700 1 --2- 172.21.15.118:0/1451592334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f8100106750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f80f40098a0 tx=0x7f80f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27c506b7519a9f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:49.611 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f80fd59a700 1 -- 172.21.15.118:0/1451592334 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f4004180 con 0x7f8100106330 2024-03-12T03:15:49.611 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f80fd59a700 1 -- 172.21.15.118:0/1451592334 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f80f40042e0 con 0x7f8100106330 2024-03-12T03:15:49.611 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f80fd59a700 1 -- 172.21.15.118:0/1451592334 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f4011a70 con 0x7f8100106330 2024-03-12T03:15:49.611 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f810488c700 1 -- 172.21.15.118:0/1451592334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 msgr2=0x7f8100106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f810488c700 1 --2- 172.21.15.118:0/1451592334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f8100106750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f80f40098a0 tx=0x7f80f4009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f810488c700 1 -- 172.21.15.118:0/1451592334 shutdown_connections 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.764+0000 7f810488c700 1 --2- 172.21.15.118:0/1451592334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f8100106750 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.765+0000 7f810488c700 1 -- 172.21.15.118:0/1451592334 >> 172.21.15.118:0/1451592334 conn(0x7f81001018b0 msgr2=0x7f8100103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.765+0000 7f810488c700 1 -- 172.21.15.118:0/1451592334 shutdown_connections 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.765+0000 7f810488c700 1 -- 172.21.15.118:0/1451592334 wait complete. 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.765+0000 7f810488c700 1 Processor -- start 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.765+0000 7f810488c700 1 -- start start 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f810488c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f810019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f810488c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810019a5e0 con 0x7f8100106330 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80fe59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f810019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80fe59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f810019a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37400/0 (socket says 172.21.15.118:37400) 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80fe59c700 1 -- 172.21.15.118:0/2523554247 learned_addr learned my addr 172.21.15.118:0/2523554247 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80fe59c700 1 -- 172.21.15.118:0/2523554247 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f4009550 con 0x7f8100106330 2024-03-12T03:15:49.612 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80fe59c700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f810019a0a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f80f4011710 tx=0x7f80f40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80ef7fe700 1 -- 172.21.15.118:0/2523554247 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f4003b10 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80ef7fe700 1 -- 172.21.15.118:0/2523554247 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f80f401b8c0 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810019a7e0 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f80ef7fe700 1 -- 172.21.15.118:0/2523554247 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f4019770 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.766+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810019ac80 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.767+0000 7f80ef7fe700 1 -- 172.21.15.118:0/2523554247 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 45378+0+0 (secure 0 0 0) 0x7f80f40199a0 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.767+0000 7f80ef7fe700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f80e80384f0 0x7f80e803a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.767+0000 7f80ef7fe700 1 -- 172.21.15.118:0/2523554247 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f80f404c6d0 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.767+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810010a760 con 0x7f8100106330 2024-03-12T03:15:49.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.767+0000 7f80fdd9b700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f80e80384f0 0x7f80e803a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:49.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.768+0000 7f80fdd9b700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f80e80384f0 0x7f80e803a9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f80f0006fd0 tx=0x7f80f0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:49.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.772+0000 7f80ef7fe700 1 -- 172.21.15.118:0/2523554247 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f80f401a3d0 con 0x7f8100106330 2024-03-12T03:15:49.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.964+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f810019edd0 con 0x7f8100106330 2024-03-12T03:15:49.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.965+0000 7f80ef7fe700 1 -- 172.21.15.118:0/2523554247 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f80f4018350 con 0x7f8100106330 2024-03-12T03:15:49.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f80e80384f0 msgr2=0x7f80e803a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:49.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f80e80384f0 0x7f80e803a9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f80f0006fd0 tx=0x7f80f0006e40 comp rx=0 tx=0).stop 2024-03-12T03:15:49.615 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 msgr2=0x7f810019a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:49.615 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f810019a0a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f80f4011710 tx=0x7f80f40047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:49.615 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 shutdown_connections 2024-03-12T03:15:49.615 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f80e80384f0 0x7f80e803a9b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:49.615 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 --2- 172.21.15.118:0/2523554247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8100106330 0x7f810019a0a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:49.615 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 >> 172.21.15.118:0/2523554247 conn(0x7f81001018b0 msgr2=0x7f8100102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:49.616 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 shutdown_connections 2024-03-12T03:15:49.616 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:48.967+0000 7f810488c700 1 -- 172.21.15.118:0/2523554247 wait complete. 2024-03-12T03:15:49.616 INFO:teuthology.orchestra.run.smithi118.stdout:Enabling cephadm module... 2024-03-12T03:15:49.994 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:49 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2523554247' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:15:51.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:50 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1430093866' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.033+0000 7f7f09343700 1 Processor -- start 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.033+0000 7f7f09343700 1 -- start start 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.033+0000 7f7f09343700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f04109200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.033+0000 7f7f09343700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f04079d90 con 0x7f7f04106e10 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f02ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f04109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f02ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f04109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37402/0 (socket says 172.21.15.118:37402) 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f02ffd700 1 -- 172.21.15.118:0/3994066033 learned_addr learned my addr 172.21.15.118:0/3994066033 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f02ffd700 1 -- 172.21.15.118:0/3994066033 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f04109740 con 0x7f7f04106e10 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f02ffd700 1 --2- 172.21.15.118:0/3994066033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f04109200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009e80 tx=0x7f7ef400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d031b1839c37008e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:51.380 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f01ffb700 1 -- 172.21.15.118:0/3994066033 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ef400e070 con 0x7f7f04106e10 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f01ffb700 1 -- 172.21.15.118:0/3994066033 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7ef400b8c0 con 0x7f7f04106e10 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.034+0000 7f7f01ffb700 1 -- 172.21.15.118:0/3994066033 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ef4004440 con 0x7f7f04106e10 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 -- 172.21.15.118:0/3994066033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 msgr2=0x7f7f04109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 --2- 172.21.15.118:0/3994066033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f04109200 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4009e80 tx=0x7f7ef400b330 comp rx=0 tx=0).stop 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 -- 172.21.15.118:0/3994066033 shutdown_connections 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 --2- 172.21.15.118:0/3994066033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f04109200 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 -- 172.21.15.118:0/3994066033 >> 172.21.15.118:0/3994066033 conn(0x7f7f04100840 msgr2=0x7f7f04102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 -- 172.21.15.118:0/3994066033 shutdown_connections 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 -- 172.21.15.118:0/3994066033 wait complete. 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.035+0000 7f7f09343700 1 Processor -- start 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f09343700 1 -- start start 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f09343700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f0419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f09343700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0419a5b0 con 0x7f7f04106e10 2024-03-12T03:15:51.381 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f02ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f0419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f02ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f0419a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37404/0 (socket says 172.21.15.118:37404) 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f02ffd700 1 -- 172.21.15.118:0/1430093866 learned_addr learned my addr 172.21.15.118:0/1430093866 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f02ffd700 1 -- 172.21.15.118:0/1430093866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4009b30 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f02ffd700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f0419a070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40099e0 tx=0x7f7ef4003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ef400e040 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f7ef400bc40 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0419a7b0 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ef40125e0 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.036+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0419acb0 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.037+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 45378+0+0 (secure 0 0 0) 0x7f7ef4019040 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.037+0000 7f7efbfff700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f7ef0038460 0x7f7ef003a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.037+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0410a460 con 0x7f7f04106e10 2024-03-12T03:15:51.382 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.037+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7ef404bab0 con 0x7f7f04106e10 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.038+0000 7f7f027fc700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f7ef0038460 0x7f7ef003a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.038+0000 7f7f027fc700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f7ef0038460 0x7f7ef003a920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7eec006fd0 tx=0x7f7eec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.042+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7ef401f080 con 0x7f7f04106e10 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.288+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7f0419d900 con 0x7f7f04106e10 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.801+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 5) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f7ef4024400 con 0x7f7f04106e10 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.803+0000 7f7efbfff700 1 -- 172.21.15.118:0/1430093866 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f7ef400bdb0 con 0x7f7f04106e10 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f7ef0038460 msgr2=0x7f7ef003a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f7ef0038460 0x7f7ef003a920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7eec006fd0 tx=0x7f7eec006e40 comp rx=0 tx=0).stop 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 msgr2=0x7f7f0419a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f0419a070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40099e0 tx=0x7f7ef4003ce0 comp rx=0 tx=0).stop 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 shutdown_connections 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f7ef0038460 0x7f7ef003a920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:51.383 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 --2- 172.21.15.118:0/1430093866 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f04106e10 0x7f7f0419a070 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:51.384 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 >> 172.21.15.118:0/1430093866 conn(0x7f7f04100840 msgr2=0x7f7f041013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:51.384 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 shutdown_connections 2024-03-12T03:15:51.384 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:50.806+0000 7f7f09343700 1 -- 172.21.15.118:0/1430093866 wait complete. 2024-03-12T03:15:52.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:51 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1430093866' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T03:15:52.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:51 smithi118 ceph-mon[29160]: mgrmap e5: smithi118.zzzuvl(active, since 13s) 2024-03-12T03:15:53.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:52 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1577198942' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:15:53.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:15:53.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-12T03:15:53.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:15:53.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "active_name": "smithi118.zzzuvl", 2024-03-12T03:15:53.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:15:53.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.539+0000 7f24c2d44700 1 Processor -- start 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.539+0000 7f24c2d44700 1 -- start start 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.539+0000 7f24c2d44700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc105620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.539+0000 7f24c2d44700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc105bf0 con 0x7f24bc105200 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.539+0000 7f24c0ae0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc105620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.539+0000 7f24c0ae0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc105620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37434/0 (socket says 172.21.15.118:37434) 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.539+0000 7f24c0ae0700 1 -- 172.21.15.118:0/3835077808 learned_addr learned my addr 172.21.15.118:0/3835077808 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.540+0000 7f24c0ae0700 1 -- 172.21.15.118:0/3835077808 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc106400 con 0x7f24bc105200 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.540+0000 7f24c0ae0700 1 --2- 172.21.15.118:0/3835077808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc105620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f24b00098a0 tx=0x7f24b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca911076c9880f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.540+0000 7f24bb7fe700 1 -- 172.21.15.118:0/3835077808 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b0004180 con 0x7f24bc105200 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.540+0000 7f24bb7fe700 1 -- 172.21.15.118:0/3835077808 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f24b00042e0 con 0x7f24bc105200 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.540+0000 7f24bb7fe700 1 -- 172.21.15.118:0/3835077808 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b0011a70 con 0x7f24bc105200 2024-03-12T03:15:53.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.540+0000 7f24c2d44700 1 -- 172.21.15.118:0/3835077808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 msgr2=0x7f24bc105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 --2- 172.21.15.118:0/3835077808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc105620 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f24b00098a0 tx=0x7f24b0009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 -- 172.21.15.118:0/3835077808 shutdown_connections 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 --2- 172.21.15.118:0/3835077808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc105620 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 -- 172.21.15.118:0/3835077808 >> 172.21.15.118:0/3835077808 conn(0x7f24bc1007a0 msgr2=0x7f24bc102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 -- 172.21.15.118:0/3835077808 shutdown_connections 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 -- 172.21.15.118:0/3835077808 wait complete. 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 Processor -- start 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 -- start start 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.541+0000 7f24c2d44700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc1994f0 con 0x7f24bc105200 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24c0ae0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24c0ae0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37436/0 (socket says 172.21.15.118:37436) 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24c0ae0700 1 -- 172.21.15.118:0/1577198942 learned_addr learned my addr 172.21.15.118:0/1577198942 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:53.474 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24c0ae0700 1 -- 172.21.15.118:0/1577198942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b0009550 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24c0ae0700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc198fb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f24b0011710 tx=0x7f24b00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24b9ffb700 1 -- 172.21.15.118:0/1577198942 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b0003b10 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24b9ffb700 1 -- 172.21.15.118:0/1577198942 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f24b001b8c0 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc1996f0 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24b9ffb700 1 -- 172.21.15.118:0/1577198942 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24b0019770 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.542+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24bc199b90 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.543+0000 7f24b9ffb700 1 -- 172.21.15.118:0/1577198942 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f24b00199a0 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.543+0000 7f24b9ffb700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 0x7f24ac03aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.543+0000 7f24b9ffb700 1 -- 172.21.15.118:0/1577198942 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f24b004c5d0 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.543+0000 7f24bbfff700 1 -- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 msgr2=0x7f24ac03aa00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1005219268 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.543+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24bc109630 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.543+0000 7f24bbfff700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 0x7f24ac03aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.548+0000 7f24b9ffb700 1 -- 172.21.15.118:0/1577198942 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f24b001f080 con 0x7f24bc105200 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.745+0000 7f24bbfff700 1 -- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 msgr2=0x7f24ac03aa00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1005219268 2024-03-12T03:15:53.475 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.745+0000 7f24bbfff700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 0x7f24ac03aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.804+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f24bc19cbc0 con 0x7f24bc105200 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.805+0000 7f24b9ffb700 1 -- 172.21.15.118:0/1577198942 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f24b001a3d0 con 0x7f24bc105200 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 msgr2=0x7f24ac03aa00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 0x7f24ac03aa00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 msgr2=0x7f24bc198fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc198fb0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f24b0011710 tx=0x7f24b00047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 shutdown_connections 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7f24ac038540 0x7f24ac03aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 --2- 172.21.15.118:0/1577198942 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24bc105200 0x7f24bc198fb0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 >> 172.21.15.118:0/1577198942 conn(0x7f24bc1007a0 msgr2=0x7f24bc1013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 shutdown_connections 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:52.807+0000 7f24c2d44700 1 -- 172.21.15.118:0/1577198942 wait complete. 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for the mgr to restart... 2024-03-12T03:15:53.476 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr epoch 5... 2024-03-12T03:15:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: Active manager daemon smithi118.zzzuvl restarted 2024-03-12T03:15:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: Activating manager daemon smithi118.zzzuvl 2024-03-12T03:15:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T03:15:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: mgrmap e6: smithi118.zzzuvl(active, starting, since 0.00317676s) 2024-03-12T03:15:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:15:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr metadata", "who": "smithi118.zzzuvl", "id": "smithi118.zzzuvl"}]: dispatch 2024-03-12T03:15:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: Manager daemon smithi118.zzzuvl is now available 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:57 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:15:58.928 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:58 smithi118 ceph-mon[29160]: Found migration_current of "None". Setting to last migration. 2024-03-12T03:15:58.928 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:58 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:15:58.928 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:58 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/trash_purge_schedule"}]: dispatch 2024-03-12T03:15:58.928 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:58 smithi118 ceph-mon[29160]: mgrmap e7: smithi118.zzzuvl(active, since 1.00537s) 2024-03-12T03:15:59.181 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.875+0000 7fbf64155700 1 Processor -- start 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.875+0000 7fbf64155700 1 -- start start 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.875+0000 7fbf64155700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c105660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.875+0000 7fbf64155700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c105c30 con 0x7fbf5c105240 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.875+0000 7fbf61ef1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c105660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.875+0000 7fbf61ef1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c105660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37450/0 (socket says 172.21.15.118:37450) 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.875+0000 7fbf61ef1700 1 -- 172.21.15.118:0/672748674 learned_addr learned my addr 172.21.15.118:0/672748674 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.876+0000 7fbf61ef1700 1 -- 172.21.15.118:0/672748674 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c106440 con 0x7fbf5c105240 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.876+0000 7fbf61ef1700 1 --2- 172.21.15.118:0/672748674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c105660 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbf580098a0 tx=0x7fbf58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278043e00dcfea91 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:59.182 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.876+0000 7fbf60eef700 1 -- 172.21.15.118:0/672748674 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf58004180 con 0x7fbf5c105240 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.876+0000 7fbf60eef700 1 -- 172.21.15.118:0/672748674 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbf580042e0 con 0x7fbf5c105240 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.876+0000 7fbf60eef700 1 -- 172.21.15.118:0/672748674 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf58011a70 con 0x7fbf5c105240 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 -- 172.21.15.118:0/672748674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 msgr2=0x7fbf5c105660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 --2- 172.21.15.118:0/672748674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c105660 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbf580098a0 tx=0x7fbf58009bb0 comp rx=0 tx=0).stop 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 -- 172.21.15.118:0/672748674 shutdown_connections 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 --2- 172.21.15.118:0/672748674 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c105660 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 -- 172.21.15.118:0/672748674 >> 172.21.15.118:0/672748674 conn(0x7fbf5c100840 msgr2=0x7fbf5c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 -- 172.21.15.118:0/672748674 shutdown_connections 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 -- 172.21.15.118:0/672748674 wait complete. 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 Processor -- start 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.877+0000 7fbf64155700 1 -- start start 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf64155700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:59.183 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf64155700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c199500 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf61ef1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf61ef1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c198fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37460/0 (socket says 172.21.15.118:37460) 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf61ef1700 1 -- 172.21.15.118:0/2559816974 learned_addr learned my addr 172.21.15.118:0/2559816974 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf61ef1700 1 -- 172.21.15.118:0/2559816974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58009550 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf61ef1700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c198fc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf58011710 tx=0x7fbf580047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf58003b10 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fbf5801b8c0 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf58019770 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c199700 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.878+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c199ba0 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.879+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 45389+0+0 (secure 0 0 0) 0x7fbf5801ba30 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.880+0000 7fbf52ffd700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.880+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 --> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf4803b0c0 con 0x7fbf480384f0 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.880+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbf5804c600 con 0x7fbf5c105240 2024-03-12T03:15:59.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.880+0000 7fbf616f0700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 msgr2=0x7fbf4803a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1005219268 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:53.880+0000 7fbf616f0700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:54.081+0000 7fbf616f0700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 msgr2=0x7fbf4803a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1005219268 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:54.081+0000 7fbf616f0700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:54.482+0000 7fbf616f0700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 msgr2=0x7fbf4803a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1005219268 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:54.482+0000 7fbf616f0700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:55.283+0000 7fbf616f0700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 msgr2=0x7fbf4803a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1005219268 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:55.283+0000 7fbf616f0700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:56.884+0000 7fbf616f0700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 msgr2=0x7fbf4803a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1005219268 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:56.884+0000 7fbf616f0700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:57.610+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mgrmap(e 6) v1 ==== 45156+0+0 (secure 0 0 0) 0x7fbf580199f0 con 0x7fbf5c105240 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:57.611+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 msgr2=0x7fbf4803a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:57.611+0000 7fbf52ffd700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.612+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 7) v1 ==== 45283+0+0 (secure 0 0 0) 0x7fbf5802c350 con 0x7fbf5c105240 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.612+0000 7fbf52ffd700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fbf4803bdd0 0x7fbf4803e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:15:59.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.612+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf4803b0c0 con 0x7fbf4803bdd0 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.612+0000 7fbf616f0700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fbf4803bdd0 0x7fbf4803e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.613+0000 7fbf616f0700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fbf4803bdd0 0x7fbf4803e1c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c003a10 tx=0x7fbf4c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.614+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fbf4803b0c0 con 0x7fbf4803bdd0 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.620+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbf5c0623e0 con 0x7fbf4803bdd0 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf52ffd700 1 -- 172.21.15.118:0/2559816974 <== mgr.14120 v2:172.21.15.118:6800/1066995478 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbf5c0623e0 con 0x7fbf4803bdd0 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fbf4803bdd0 msgr2=0x7fbf4803e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fbf4803bdd0 0x7fbf4803e1c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c003a10 tx=0x7fbf4c009290 comp rx=0 tx=0).stop 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 msgr2=0x7fbf5c198fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c198fc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbf58011710 tx=0x7fbf580047c0 comp rx=0 tx=0).stop 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 shutdown_connections 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fbf4803bdd0 0x7fbf4803e1c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbf5c105240 0x7fbf5c198fc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 --2- 172.21.15.118:0/2559816974 >> [v2:172.21.15.118:6800/1005219268,v1:172.21.15.118:6801/1005219268] conn(0x7fbf480384f0 0x7fbf4803a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:15:59.186 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 >> 172.21.15.118:0/2559816974 conn(0x7fbf5c100840 msgr2=0x7fbf5c101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:15:59.187 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 shutdown_connections 2024-03-12T03:15:59.187 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:58.621+0000 7fbf64155700 1 -- 172.21.15.118:0/2559816974 wait complete. 2024-03-12T03:15:59.187 INFO:teuthology.orchestra.run.smithi118.stdout:mgr epoch 5 is available 2024-03-12T03:15:59.187 INFO:teuthology.orchestra.run.smithi118.stdout:Setting orchestrator backend to cephadm... 2024-03-12T03:16:00.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:59 smithi118 ceph-mon[29160]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:16:00.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:59 smithi118 ceph-mon[29160]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:16:00.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:59 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:00.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:59 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:00.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:15:59 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:00.435 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb3d92700 1 Processor -- start 2024-03-12T03:16:00.435 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb3d92700 1 -- start start 2024-03-12T03:16:00.435 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb3d92700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac104a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:00.435 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb3d92700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac105000 con 0x7f0bac104610 2024-03-12T03:16:00.435 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb1b2e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb1b2e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac104a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48832/0 (socket says 172.21.15.118:48832) 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb1b2e700 1 -- 172.21.15.118:0/2389541910 learned_addr learned my addr 172.21.15.118:0/2389541910 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.594+0000 7f0bb1b2e700 1 -- 172.21.15.118:0/2389541910 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac105330 con 0x7f0bac104610 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb1b2e700 1 --2- 172.21.15.118:0/2389541910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac104a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80098a0 tx=0x7f0ba8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfdaaba91b10d19b server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb0b2c700 1 -- 172.21.15.118:0/2389541910 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba8004180 con 0x7f0bac104610 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb0b2c700 1 -- 172.21.15.118:0/2389541910 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0ba80042e0 con 0x7f0bac104610 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb0b2c700 1 -- 172.21.15.118:0/2389541910 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba8011a70 con 0x7f0bac104610 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb3d92700 1 -- 172.21.15.118:0/2389541910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 msgr2=0x7f0bac104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb3d92700 1 --2- 172.21.15.118:0/2389541910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac104a30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0ba80098a0 tx=0x7f0ba8009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb3d92700 1 -- 172.21.15.118:0/2389541910 shutdown_connections 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb3d92700 1 --2- 172.21.15.118:0/2389541910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac104a30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb3d92700 1 -- 172.21.15.118:0/2389541910 >> 172.21.15.118:0/2389541910 conn(0x7f0bac0fffc0 msgr2=0x7f0bac1023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb3d92700 1 -- 172.21.15.118:0/2389541910 shutdown_connections 2024-03-12T03:16:00.436 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.595+0000 7f0bb3d92700 1 -- 172.21.15.118:0/2389541910 wait complete. 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb3d92700 1 Processor -- start 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb3d92700 1 -- start start 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb3d92700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb3d92700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac1a0ba0 con 0x7f0bac104610 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb1b2e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb1b2e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48838/0 (socket says 172.21.15.118:48838) 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb1b2e700 1 -- 172.21.15.118:0/12377714 learned_addr learned my addr 172.21.15.118:0/12377714 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.596+0000 7f0bb1b2e700 1 -- 172.21.15.118:0/12377714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8009550 con 0x7f0bac104610 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.597+0000 7f0bb1b2e700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac1a0660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8011710 tx=0x7f0ba80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.597+0000 7f0ba2ffd700 1 -- 172.21.15.118:0/12377714 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba8003b10 con 0x7f0bac104610 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.597+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac1a0da0 con 0x7f0bac104610 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.597+0000 7f0ba2ffd700 1 -- 172.21.15.118:0/12377714 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f0ba801b8c0 con 0x7f0bac104610 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.597+0000 7f0ba2ffd700 1 -- 172.21.15.118:0/12377714 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba8019770 con 0x7f0bac104610 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.597+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac1a1240 con 0x7f0bac104610 2024-03-12T03:16:00.437 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.598+0000 7f0ba2ffd700 1 -- 172.21.15.118:0/12377714 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 7) v1 ==== 45283+0+0 (secure 0 0 0) 0x7f0ba801ba30 con 0x7f0bac104610 2024-03-12T03:16:00.438 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.598+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bac19a670 con 0x7f0bac104610 2024-03-12T03:16:00.438 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.598+0000 7f0ba2ffd700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0b980383d0 0x7f0b9803a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:00.438 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.598+0000 7f0ba2ffd700 1 -- 172.21.15.118:0/12377714 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0ba804c490 con 0x7f0bac104610 2024-03-12T03:16:00.438 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.598+0000 7f0bb132d700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0b980383d0 0x7f0b9803a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:00.438 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.598+0000 7f0bb132d700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0b980383d0 0x7f0b9803a890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c006fd0 tx=0x7f0b9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.603+0000 7f0ba2ffd700 1 -- 172.21.15.118:0/12377714 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0ba802cce0 con 0x7f0bac104610 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.812+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f0bac0611f0 con 0x7f0b980383d0 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.819+0000 7f0ba2ffd700 1 -- 172.21.15.118:0/12377714 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0bac0611f0 con 0x7f0b980383d0 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0b980383d0 msgr2=0x7f0b9803a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0b980383d0 0x7f0b9803a890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c006fd0 tx=0x7f0b9c006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 msgr2=0x7f0bac1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac1a0660 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ba8011710 tx=0x7f0ba80047c0 comp rx=0 tx=0).stop 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 shutdown_connections 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0b980383d0 0x7f0b9803a890 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 --2- 172.21.15.118:0/12377714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0bac104610 0x7f0bac1a0660 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 >> 172.21.15.118:0/12377714 conn(0x7f0bac0fffc0 msgr2=0x7f0bac10ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 shutdown_connections 2024-03-12T03:16:00.439 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:15:59.821+0000 7f0bb3d92700 1 -- 172.21.15.118:0/12377714 wait complete. 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: [12/Mar/2024:03:15:59] ENGINE Bus STARTING 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: [12/Mar/2024:03:15:59] ENGINE Serving on http://172.21.15.118:8765 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: [12/Mar/2024:03:15:59] ENGINE Serving on https://172.21.15.118:7150 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: [12/Mar/2024:03:15:59] ENGINE Bus STARTED 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:01.052 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:00 smithi118 ceph-mon[29160]: mgrmap e8: smithi118.zzzuvl(active, since 2s) 2024-03-12T03:16:01.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-12T03:16:01.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.843+0000 7febdd188700 1 Processor -- start 2024-03-12T03:16:01.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.843+0000 7febdd188700 1 -- start start 2024-03-12T03:16:01.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febdd188700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd8105660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febdd188700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd8105c30 con 0x7febd8105240 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd6d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd8105660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd6d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd8105660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48850/0 (socket says 172.21.15.118:48850) 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd6d9d700 1 -- 172.21.15.118:0/1337441795 learned_addr learned my addr 172.21.15.118:0/1337441795 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd6d9d700 1 -- 172.21.15.118:0/1337441795 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd8106440 con 0x7febd8105240 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd6d9d700 1 --2- 172.21.15.118:0/1337441795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd8105660 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7febc80098a0 tx=0x7febc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e2cc31840ebd05 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd5d9b700 1 -- 172.21.15.118:0/1337441795 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8004180 con 0x7febd8105240 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd5d9b700 1 -- 172.21.15.118:0/1337441795 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7febc80042e0 con 0x7febd8105240 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.844+0000 7febd5d9b700 1 -- 172.21.15.118:0/1337441795 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8011a70 con 0x7febd8105240 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.845+0000 7febdd188700 1 -- 172.21.15.118:0/1337441795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 msgr2=0x7febd8105660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.845+0000 7febdd188700 1 --2- 172.21.15.118:0/1337441795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd8105660 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7febc80098a0 tx=0x7febc8009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:01.626 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.845+0000 7febdd188700 1 -- 172.21.15.118:0/1337441795 shutdown_connections 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.845+0000 7febdd188700 1 --2- 172.21.15.118:0/1337441795 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd8105660 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.845+0000 7febdd188700 1 -- 172.21.15.118:0/1337441795 >> 172.21.15.118:0/1337441795 conn(0x7febd8100840 msgr2=0x7febd8102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.845+0000 7febdd188700 1 -- 172.21.15.118:0/1337441795 shutdown_connections 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.845+0000 7febdd188700 1 -- 172.21.15.118:0/1337441795 wait complete. 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febdd188700 1 Processor -- start 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febdd188700 1 -- start start 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febdd188700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febdd188700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd819a650 con 0x7febd8105240 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febd6d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:01.627 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febd6d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48852/0 (socket says 172.21.15.118:48852) 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febd6d9d700 1 -- 172.21.15.118:0/3701018526 learned_addr learned my addr 172.21.15.118:0/3701018526 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febd6d9d700 1 -- 172.21.15.118:0/3701018526 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc8009550 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.846+0000 7febd6d9d700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd819a110 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febc8011710 tx=0x7febc80047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.847+0000 7febbffff700 1 -- 172.21.15.118:0/3701018526 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8003b10 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.847+0000 7febbffff700 1 -- 172.21.15.118:0/3701018526 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7febc801b8c0 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.847+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febd819a850 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.847+0000 7febbffff700 1 -- 172.21.15.118:0/3701018526 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febc8019710 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.847+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febd819acf0 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.847+0000 7febbffff700 1 -- 172.21.15.118:0/3701018526 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7febc80198c0 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.848+0000 7febbffff700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7febc0038500 0x7febc003a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.848+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd8194260 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.848+0000 7febbffff700 1 -- 172.21.15.118:0/3701018526 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7febc804c7b0 con 0x7febd8105240 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.848+0000 7febd659c700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7febc0038500 0x7febc003a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:01.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.848+0000 7febd659c700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7febc0038500 0x7febc003a9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febcc006fd0 tx=0x7febcc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:00.852+0000 7febbffff700 1 -- 172.21.15.118:0/3701018526 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7febc8018b80 con 0x7febd8105240 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.049+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7febd80611f0 con 0x7febc0038500 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.050+0000 7febbffff700 1 -- 172.21.15.118:0/3701018526 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7febd80611f0 con 0x7febc0038500 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.052+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7febc0038500 msgr2=0x7febc003a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.052+0000 7febdd188700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7febc0038500 0x7febc003a9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7febcc006fd0 tx=0x7febcc006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 msgr2=0x7febd819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd819a110 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7febc8011710 tx=0x7febc80047c0 comp rx=0 tx=0).stop 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 shutdown_connections 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7febc0038500 0x7febc003a9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 --2- 172.21.15.118:0/3701018526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7febd8105240 0x7febd819a110 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 >> 172.21.15.118:0/3701018526 conn(0x7febd8100840 msgr2=0x7febd8101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 shutdown_connections 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:01.053+0000 7febdd188700 1 -- 172.21.15.118:0/3701018526 wait complete. 2024-03-12T03:16:01.629 INFO:teuthology.orchestra.run.smithi118.stdout:Generating ssh key... 2024-03-12T03:16:02.267 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:01 smithi118 ceph-mon[29160]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa910c1700 1 Processor -- start 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa910c1700 1 -- start start 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa910c1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c106740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa910c1700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c106d10 con 0x7ffa8c106320 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa8ad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa8ad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48860/0 (socket says 172.21.15.118:48860) 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa8ad9d700 1 -- 172.21.15.118:0/2190229106 learned_addr learned my addr 172.21.15.118:0/2190229106 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa8ad9d700 1 -- 172.21.15.118:0/2190229106 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa8c107520 con 0x7ffa8c106320 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.062+0000 7ffa8ad9d700 1 --2- 172.21.15.118:0/2190229106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c106740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c0098a0 tx=0x7ffa7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a1c16428b96e43 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa89d9b700 1 -- 172.21.15.118:0/2190229106 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7c004180 con 0x7ffa8c106320 2024-03-12T03:16:03.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa89d9b700 1 -- 172.21.15.118:0/2190229106 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ffa7c0042e0 con 0x7ffa8c106320 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa89d9b700 1 -- 172.21.15.118:0/2190229106 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7c011a70 con 0x7ffa8c106320 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa910c1700 1 -- 172.21.15.118:0/2190229106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 msgr2=0x7ffa8c106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa910c1700 1 --2- 172.21.15.118:0/2190229106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c106740 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c0098a0 tx=0x7ffa7c009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa910c1700 1 -- 172.21.15.118:0/2190229106 shutdown_connections 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa910c1700 1 --2- 172.21.15.118:0/2190229106 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c106740 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa910c1700 1 -- 172.21.15.118:0/2190229106 >> 172.21.15.118:0/2190229106 conn(0x7ffa8c101880 msgr2=0x7ffa8c103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa910c1700 1 -- 172.21.15.118:0/2190229106 shutdown_connections 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.063+0000 7ffa910c1700 1 -- 172.21.15.118:0/2190229106 wait complete. 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.064+0000 7ffa910c1700 1 Processor -- start 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.064+0000 7ffa910c1700 1 -- start start 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.064+0000 7ffa910c1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.064+0000 7ffa910c1700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c19a5f0 con 0x7ffa8c106320 2024-03-12T03:16:03.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.064+0000 7ffa8ad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.064+0000 7ffa8ad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c19a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48866/0 (socket says 172.21.15.118:48866) 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.064+0000 7ffa8ad9d700 1 -- 172.21.15.118:0/114361275 learned_addr learned my addr 172.21.15.118:0/114361275 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.065+0000 7ffa8ad9d700 1 -- 172.21.15.118:0/114361275 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7c009550 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.065+0000 7ffa8ad9d700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c19a0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c011710 tx=0x7ffa7c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.065+0000 7ffa73fff700 1 -- 172.21.15.118:0/114361275 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7c003b10 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.065+0000 7ffa73fff700 1 -- 172.21.15.118:0/114361275 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7ffa7c01b8c0 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.065+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c19a7f0 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.065+0000 7ffa73fff700 1 -- 172.21.15.118:0/114361275 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa7c019710 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.065+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8c19ac90 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.066+0000 7ffa73fff700 1 -- 172.21.15.118:0/114361275 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7ffa7c0198c0 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.066+0000 7ffa73fff700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7ffa74038500 0x7ffa7403a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.066+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8c10a730 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.066+0000 7ffa73fff700 1 -- 172.21.15.118:0/114361275 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffa7c04c820 con 0x7ffa8c106320 2024-03-12T03:16:03.235 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.066+0000 7ffa8a59c700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7ffa74038500 0x7ffa7403a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.066+0000 7ffa8a59c700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7ffa74038500 0x7ffa7403a9c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa80006fd0 tx=0x7ffa80006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.071+0000 7ffa73fff700 1 -- 172.21.15.118:0/114361275 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffa7c019bc0 con 0x7ffa8c106320 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.265+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa8c102be0 con 0x7ffa74038500 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.659+0000 7ffa73fff700 1 -- 172.21.15.118:0/114361275 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffa8c102be0 con 0x7ffa74038500 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7ffa74038500 msgr2=0x7ffa7403a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7ffa74038500 0x7ffa7403a9c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa80006fd0 tx=0x7ffa80006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 msgr2=0x7ffa8c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c19a0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c011710 tx=0x7ffa7c0047c0 comp rx=0 tx=0).stop 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 shutdown_connections 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7ffa74038500 0x7ffa7403a9c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 --2- 172.21.15.118:0/114361275 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffa8c106320 0x7ffa8c19a0b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 >> 172.21.15.118:0/114361275 conn(0x7ffa8c101880 msgr2=0x7ffa8c1024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 shutdown_connections 2024-03-12T03:16:03.236 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:02.661+0000 7ffa910c1700 1 -- 172.21.15.118:0/114361275 wait complete. 2024-03-12T03:16:03.912 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:03 smithi118 ceph-mon[29160]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:03.912 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:03 smithi118 ceph-mon[29160]: Generating ssh key... 2024-03-12T03:16:03.912 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:03 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:03.912 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:03 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:04.707 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ZpAcWYiwBVeEkSS1JXdHkfbiOhNSplrlpqUzMe2B2dBA6kvvGYVParX72/s/yVTXBrASTWD5CVxc60ioJfitnieEHayZvLfzYsejyJzBsHQ34Sigs36PgTyzG7Cbt23+gF8fJyN4p8DomI09iTgeLFk3nxJQP0/kNCnhXWY92gIy1qfyyJhl7c8iMOlG646zWrXABNBCD55oPtiaCS2B1d8I0ZAU75iCZesrkXVOeqKXCLIHzeXwBzDdzp0Cmf6Nprwup4XDr4I77nFzU+FkRYf5IVN/ybQC/oPm/eE8GN0T//uQEVrELRGVg/KItwBzg+IzZEfHkpsAgkFJRu+G6JAjly9cGl0wF5/caHuir0LlHgegZK/B+kIB+yugFbg7ose1VqNMs3CogY/2jnn/r16oJcvB+NnEOs0gmL2YnGNKMcD8QFSGEjpF3B2OztAC8q2zjLYMUEkIsQ3/a6JPq1GLIap9cDoeJJ+cpeqQmj/mTaIzxdeuxa0z4qSPneE= ceph-9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:16:04.707 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.931+0000 7feaaa310700 1 Processor -- start 2024-03-12T03:16:04.707 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.931+0000 7feaaa310700 1 -- start start 2024-03-12T03:16:04.707 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.931+0000 7feaaa310700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa41091a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:04.707 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.931+0000 7feaaa310700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa4074540 con 0x7feaa4106db0 2024-03-12T03:16:04.707 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa41091a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa41091a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48874/0 (socket says 172.21.15.118:48874) 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa3fff700 1 -- 172.21.15.118:0/2253834214 learned_addr learned my addr 172.21.15.118:0/2253834214 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa3fff700 1 -- 172.21.15.118:0/2253834214 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa41096e0 con 0x7feaa4106db0 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa3fff700 1 --2- 172.21.15.118:0/2253834214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa41091a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea980098a0 tx=0x7fea98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5a7d4f5449b155 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa2ffd700 1 -- 172.21.15.118:0/2253834214 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98004180 con 0x7feaa4106db0 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa2ffd700 1 -- 172.21.15.118:0/2253834214 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fea980042e0 con 0x7feaa4106db0 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.932+0000 7feaa2ffd700 1 -- 172.21.15.118:0/2253834214 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98011a70 con 0x7feaa4106db0 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 -- 172.21.15.118:0/2253834214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 msgr2=0x7feaa41091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 --2- 172.21.15.118:0/2253834214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa41091a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fea980098a0 tx=0x7fea98009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 -- 172.21.15.118:0/2253834214 shutdown_connections 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 --2- 172.21.15.118:0/2253834214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa41091a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 -- 172.21.15.118:0/2253834214 >> 172.21.15.118:0/2253834214 conn(0x7feaa41007a0 msgr2=0x7feaa4102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:04.708 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 -- 172.21.15.118:0/2253834214 shutdown_connections 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 -- 172.21.15.118:0/2253834214 wait complete. 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.933+0000 7feaaa310700 1 Processor -- start 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaaa310700 1 -- start start 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaaa310700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaaa310700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa419a5a0 con 0x7feaa4106db0 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaa3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa419a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48878/0 (socket says 172.21.15.118:48878) 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaa3fff700 1 -- 172.21.15.118:0/2527252177 learned_addr learned my addr 172.21.15.118:0/2527252177 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaa3fff700 1 -- 172.21.15.118:0/2527252177 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea98009550 con 0x7feaa4106db0 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaa3fff700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa419a060 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea98011a20 tx=0x7fea980047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.934+0000 7feaa17fa700 1 -- 172.21.15.118:0/2527252177 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea98003b10 con 0x7feaa4106db0 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.935+0000 7feaa17fa700 1 -- 172.21.15.118:0/2527252177 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7fea9801b8c0 con 0x7feaa4106db0 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.935+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa419a7a0 con 0x7feaa4106db0 2024-03-12T03:16:04.709 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.935+0000 7feaa17fa700 1 -- 172.21.15.118:0/2527252177 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea980237b0 con 0x7feaa4106db0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.935+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa419ac40 con 0x7feaa4106db0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.935+0000 7feaa17fa700 1 -- 172.21.15.118:0/2527252177 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7fea98023a30 con 0x7feaa4106db0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.935+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa410a430 con 0x7feaa4106db0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.936+0000 7feaa17fa700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fea8c0384b0 0x7fea8c03a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.936+0000 7feaa17fa700 1 -- 172.21.15.118:0/2527252177 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fea9804c8c0 con 0x7feaa4106db0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.936+0000 7feaa37fe700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fea8c0384b0 0x7fea8c03a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.936+0000 7feaa37fe700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fea8c0384b0 0x7fea8c03a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fea94006fd0 tx=0x7fea94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:03.941+0000 7feaa17fa700 1 -- 172.21.15.118:0/2527252177 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fea98018760 con 0x7feaa4106db0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.132+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7feaa4101a30 con 0x7fea8c0384b0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.133+0000 7feaa17fa700 1 -- 172.21.15.118:0/2527252177 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7feaa4101a30 con 0x7fea8c0384b0 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.134+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fea8c0384b0 msgr2=0x7fea8c03a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:04.710 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.134+0000 7feaaa310700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fea8c0384b0 0x7fea8c03a970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fea94006fd0 tx=0x7fea94006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.134+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 msgr2=0x7feaa419a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.134+0000 7feaaa310700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa419a060 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fea98011a20 tx=0x7fea980047c0 comp rx=0 tx=0).stop 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.135+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 shutdown_connections 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.135+0000 7feaaa310700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fea8c0384b0 0x7fea8c03a970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.135+0000 7feaaa310700 1 --2- 172.21.15.118:0/2527252177 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaa4106db0 0x7feaa419a060 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.135+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 >> 172.21.15.118:0/2527252177 conn(0x7feaa41007a0 msgr2=0x7feaa4101320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.135+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 shutdown_connections 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:04.135+0000 7feaaa310700 1 -- 172.21.15.118:0/2527252177 wait complete. 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:Adding key to root@localhost authorized_keys... 2024-03-12T03:16:04.711 INFO:teuthology.orchestra.run.smithi118.stdout:Adding host smithi118... 2024-03-12T03:16:05.002 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:04 smithi118 ceph-mon[29160]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:06.183 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:05 smithi118 ceph-mon[29160]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi118", "addr": "172.21.15.118", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:07.266 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:06 smithi118 ceph-mon[29160]: Deploying cephadm binary to smithi118 2024-03-12T03:16:08.441 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Added host 'smithi118' with addr '172.21.15.118' 2024-03-12T03:16:08.441 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.101+0000 7f5e4bb67700 1 Processor -- start 2024-03-12T03:16:08.441 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.101+0000 7f5e4bb67700 1 -- start start 2024-03-12T03:16:08.441 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.101+0000 7f5e4bb67700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44106c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:08.441 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.101+0000 7f5e4bb67700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e441071d0 con 0x7f5e441067e0 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e49903700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44106c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e49903700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44106c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35012/0 (socket says 172.21.15.118:35012) 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e49903700 1 -- 172.21.15.118:0/909744738 learned_addr learned my addr 172.21.15.118:0/909744738 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e49903700 1 -- 172.21.15.118:0/909744738 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e441053f0 con 0x7f5e441067e0 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e49903700 1 --2- 172.21.15.118:0/909744738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44106c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009e80 tx=0x7f5e4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a134ca3a1cc0d550 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e48901700 1 -- 172.21.15.118:0/909744738 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e4000e070 con 0x7f5e441067e0 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e48901700 1 -- 172.21.15.118:0/909744738 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5e4000b8c0 con 0x7f5e441067e0 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.102+0000 7f5e48901700 1 -- 172.21.15.118:0/909744738 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e40004440 con 0x7f5e441067e0 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 -- 172.21.15.118:0/909744738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 msgr2=0x7f5e44106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 --2- 172.21.15.118:0/909744738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44106c00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5e40009e80 tx=0x7f5e4000b330 comp rx=0 tx=0).stop 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 -- 172.21.15.118:0/909744738 shutdown_connections 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 --2- 172.21.15.118:0/909744738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44106c00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:08.442 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 -- 172.21.15.118:0/909744738 >> 172.21.15.118:0/909744738 conn(0x7f5e440fff20 msgr2=0x7f5e44102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 -- 172.21.15.118:0/909744738 shutdown_connections 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 -- 172.21.15.118:0/909744738 wait complete. 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 Processor -- start 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.103+0000 7f5e4bb67700 1 -- start start 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e4bb67700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44104d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e4bb67700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e44105250 con 0x7f5e441067e0 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e49903700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44104d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e49903700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44104d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35028/0 (socket says 172.21.15.118:35028) 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e49903700 1 -- 172.21.15.118:0/3350219554 learned_addr learned my addr 172.21.15.118:0/3350219554 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e49903700 1 -- 172.21.15.118:0/3350219554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e40009b30 con 0x7f5e441067e0 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e49903700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44104d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5e400099e0 tx=0x7f5e40003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e3affd700 1 -- 172.21.15.118:0/3350219554 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e4000e040 con 0x7f5e441067e0 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e3affd700 1 -- 172.21.15.118:0/3350219554 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f5e4000bc40 con 0x7f5e441067e0 2024-03-12T03:16:08.443 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.104+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4419cdc0 con 0x7f5e441067e0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.105+0000 7f5e3affd700 1 -- 172.21.15.118:0/3350219554 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e400125e0 con 0x7f5e441067e0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.105+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4419d260 con 0x7f5e441067e0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.105+0000 7f5e3affd700 1 -- 172.21.15.118:0/3350219554 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f5e40019040 con 0x7f5e441067e0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.105+0000 7f5e3affd700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f5e30038160 0x7f5e3003a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.105+0000 7f5e3affd700 1 -- 172.21.15.118:0/3350219554 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5e4004f630 con 0x7f5e441067e0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.105+0000 7f5e49102700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f5e30038160 0x7f5e3003a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.106+0000 7f5e49102700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f5e30038160 0x7f5e3003a620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e34006fd0 tx=0x7f5e34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.106+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e440623e0 con 0x7f5e441067e0 2024-03-12T03:16:08.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.110+0000 7f5e3affd700 1 -- 172.21.15.118:0/3350219554 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5e4001f020 con 0x7f5e441067e0 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:05.305+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi118", "addr": "172.21.15.118", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e441011f0 con 0x7f5e30038160 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.870+0000 7f5e3affd700 1 -- 172.21.15.118:0/3350219554 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5e441011f0 con 0x7f5e30038160 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f5e30038160 msgr2=0x7f5e3003a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f5e30038160 0x7f5e3003a620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5e34006fd0 tx=0x7f5e34006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 msgr2=0x7f5e44104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44104d10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5e400099e0 tx=0x7f5e40003d00 comp rx=0 tx=0).stop 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 shutdown_connections 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f5e30038160 0x7f5e3003a620 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 --2- 172.21.15.118:0/3350219554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e441067e0 0x7f5e44104d10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 >> 172.21.15.118:0/3350219554 conn(0x7f5e440fff20 msgr2=0x7f5e44100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 shutdown_connections 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:07.873+0000 7f5e4bb67700 1 -- 172.21.15.118:0/3350219554 wait complete. 2024-03-12T03:16:08.445 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying mon service with default placement... 2024-03-12T03:16:09.056 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:08 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:09.056 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:08 smithi118 ceph-mon[29160]: Added host smithi118 2024-03-12T03:16:09.056 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:08 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.834+0000 7f678a61b700 1 Processor -- start 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.834+0000 7f678a61b700 1 -- start start 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f678a61b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f6784105620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f678a61b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6784105bf0 con 0x7f6784105200 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6783fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f6784105620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6783fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f6784105620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35038/0 (socket says 172.21.15.118:35038) 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6783fff700 1 -- 172.21.15.118:0/842685903 learned_addr learned my addr 172.21.15.118:0/842685903 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6783fff700 1 -- 172.21.15.118:0/842685903 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6784106400 con 0x7f6784105200 2024-03-12T03:16:09.751 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6783fff700 1 --2- 172.21.15.118:0/842685903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f6784105620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f67740098a0 tx=0x7f6774009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992870e8777a055a server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6782ffd700 1 -- 172.21.15.118:0/842685903 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6774004180 con 0x7f6784105200 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6782ffd700 1 -- 172.21.15.118:0/842685903 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f67740042e0 con 0x7f6784105200 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.835+0000 7f6782ffd700 1 -- 172.21.15.118:0/842685903 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6774011a70 con 0x7f6784105200 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 -- 172.21.15.118:0/842685903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 msgr2=0x7f6784105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 --2- 172.21.15.118:0/842685903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f6784105620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f67740098a0 tx=0x7f6774009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 -- 172.21.15.118:0/842685903 shutdown_connections 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 --2- 172.21.15.118:0/842685903 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f6784105620 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 -- 172.21.15.118:0/842685903 >> 172.21.15.118:0/842685903 conn(0x7f67841007a0 msgr2=0x7f6784102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 -- 172.21.15.118:0/842685903 shutdown_connections 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 -- 172.21.15.118:0/842685903 wait complete. 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.836+0000 7f678a61b700 1 Processor -- start 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f678a61b700 1 -- start start 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f678a61b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f678419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:09.752 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f678a61b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678419a610 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f6783fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f678419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f6783fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f678419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35044/0 (socket says 172.21.15.118:35044) 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f6783fff700 1 -- 172.21.15.118:0/2053617713 learned_addr learned my addr 172.21.15.118:0/2053617713 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f6783fff700 1 -- 172.21.15.118:0/2053617713 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6774009550 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f6783fff700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f678419a0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6774011710 tx=0x7f67740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f67817fa700 1 -- 172.21.15.118:0/2053617713 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6774003b10 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.837+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678419a810 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.838+0000 7f67817fa700 1 -- 172.21.15.118:0/2053617713 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f677401b8c0 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.838+0000 7f67817fa700 1 -- 172.21.15.118:0/2053617713 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6774019710 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.838+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678419acb0 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.838+0000 7f67817fa700 1 -- 172.21.15.118:0/2053617713 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f6774019910 con 0x7f6784105200 2024-03-12T03:16:09.753 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.838+0000 7f67817fa700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f67700384b0 0x7f677003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.839+0000 7f67817fa700 1 -- 172.21.15.118:0/2053617713 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f677404cdf0 con 0x7f6784105200 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.839+0000 7f67837fe700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f67700384b0 0x7f677003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.839+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67840623e0 con 0x7f6784105200 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.839+0000 7f67837fe700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f67700384b0 0x7f677003a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f676c006fd0 tx=0x7f676c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:08.844+0000 7f67817fa700 1 -- 172.21.15.118:0/2053617713 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6774019c10 con 0x7f6784105200 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.072+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f6784101b00 con 0x7f67700384b0 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.076+0000 7f67817fa700 1 -- 172.21.15.118:0/2053617713 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6784101b00 con 0x7f67700384b0 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f67700384b0 msgr2=0x7f677003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f67700384b0 0x7f677003a970 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f676c006fd0 tx=0x7f676c006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 msgr2=0x7f678419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f678419a0d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6774011710 tx=0x7f67740047c0 comp rx=0 tx=0).stop 2024-03-12T03:16:09.754 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 shutdown_connections 2024-03-12T03:16:09.755 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f67700384b0 0x7f677003a970 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:09.755 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 --2- 172.21.15.118:0/2053617713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6784105200 0x7f678419a0d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:09.755 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 >> 172.21.15.118:0/2053617713 conn(0x7f67841007a0 msgr2=0x7f67841013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:09.755 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 shutdown_connections 2024-03-12T03:16:09.755 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:09.078+0000 7f678a61b700 1 -- 172.21.15.118:0/2053617713 wait complete. 2024-03-12T03:16:09.755 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying mgr service with default placement... 2024-03-12T03:16:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:10 smithi118 ceph-mon[29160]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:10 smithi118 ceph-mon[29160]: Saving service mon spec with placement count:5 2024-03-12T03:16:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:10 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:11.671 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:11 smithi118 ceph-mon[29160]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:11.671 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:11 smithi118 ceph-mon[29160]: Saving service mgr spec with placement count:2 2024-03-12T03:16:11.671 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:11 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:11.671 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:11 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:11.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-12T03:16:11.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.158+0000 7f2e3ddc1700 1 Processor -- start 2024-03-12T03:16:11.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.158+0000 7f2e3ddc1700 1 -- start start 2024-03-12T03:16:11.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.158+0000 7f2e3ddc1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e381091e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:11.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.158+0000 7f2e3ddc1700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e38079db0 con 0x7f2e38106df0 2024-03-12T03:16:11.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.158+0000 7f2e377fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e381091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:11.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.158+0000 7f2e377fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e381091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35054/0 (socket says 172.21.15.118:35054) 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.158+0000 7f2e377fe700 1 -- 172.21.15.118:0/3338402236 learned_addr learned my addr 172.21.15.118:0/3338402236 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.159+0000 7f2e377fe700 1 -- 172.21.15.118:0/3338402236 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e38109720 con 0x7f2e38106df0 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.159+0000 7f2e377fe700 1 --2- 172.21.15.118:0/3338402236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e381091e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2e280098a0 tx=0x7f2e28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245acaae77f68a88 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.159+0000 7f2e367fc700 1 -- 172.21.15.118:0/3338402236 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e28004180 con 0x7f2e38106df0 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.159+0000 7f2e367fc700 1 -- 172.21.15.118:0/3338402236 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2e280042e0 con 0x7f2e38106df0 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.159+0000 7f2e367fc700 1 -- 172.21.15.118:0/3338402236 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e28011a70 con 0x7f2e38106df0 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/3338402236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 msgr2=0x7f2e381091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 --2- 172.21.15.118:0/3338402236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e381091e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2e280098a0 tx=0x7f2e28009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/3338402236 shutdown_connections 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 --2- 172.21.15.118:0/3338402236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e381091e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/3338402236 >> 172.21.15.118:0/3338402236 conn(0x7f2e38100820 msgr2=0x7f2e38102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/3338402236 shutdown_connections 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/3338402236 wait complete. 2024-03-12T03:16:11.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 Processor -- start 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.160+0000 7f2e3ddc1700 1 -- start start 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e3ddc1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e3819a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e3ddc1700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e3819a590 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e377fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e3819a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e377fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e3819a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35064/0 (socket says 172.21.15.118:35064) 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e377fe700 1 -- 172.21.15.118:0/128235274 learned_addr learned my addr 172.21.15.118:0/128235274 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e377fe700 1 -- 172.21.15.118:0/128235274 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e28009550 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e377fe700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e3819a050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e28011a20 tx=0x7f2e280047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e34ff9700 1 -- 172.21.15.118:0/128235274 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e28003b10 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e3819a790 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e34ff9700 1 -- 172.21.15.118:0/128235274 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1042+0+0 (secure 0 0 0) 0x7f2e2801b8c0 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e34ff9700 1 -- 172.21.15.118:0/128235274 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e280237b0 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.161+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e3819ac30 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.162+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e3810a470 con 0x7f2e38106df0 2024-03-12T03:16:11.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.162+0000 7f2e34ff9700 1 -- 172.21.15.118:0/128235274 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f2e28023a30 con 0x7f2e38106df0 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.163+0000 7f2e34ff9700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f2e200384b0 0x7f2e2003a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.163+0000 7f2e34ff9700 1 -- 172.21.15.118:0/128235274 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2e2804c8c0 con 0x7f2e38106df0 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.163+0000 7f2e36ffd700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f2e200384b0 0x7f2e2003a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.163+0000 7f2e36ffd700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f2e200384b0 0x7f2e2003a970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c006fd0 tx=0x7f2e2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.167+0000 7f2e34ff9700 1 -- 172.21.15.118:0/128235274 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2e28018350 con 0x7f2e38106df0 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.374+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e38101a70 con 0x7f2e200384b0 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.378+0000 7f2e34ff9700 1 -- 172.21.15.118:0/128235274 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2e38101a70 con 0x7f2e200384b0 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f2e200384b0 msgr2=0x7f2e2003a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f2e200384b0 0x7f2e2003a970 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c006fd0 tx=0x7f2e2c006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 msgr2=0x7f2e3819a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e3819a050 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2e28011a20 tx=0x7f2e280047c0 comp rx=0 tx=0).stop 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 shutdown_connections 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f2e200384b0 0x7f2e2003a970 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 --2- 172.21.15.118:0/128235274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2e38106df0 0x7f2e3819a050 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:11.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.380+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 >> 172.21.15.118:0/128235274 conn(0x7f2e38100820 msgr2=0x7f2e38101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:11.682 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.381+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 shutdown_connections 2024-03-12T03:16:11.682 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:10.381+0000 7f2e3ddc1700 1 -- 172.21.15.118:0/128235274 wait complete. 2024-03-12T03:16:11.682 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying crash service with default placement... 2024-03-12T03:16:12.878 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.109+0000 7f3233b56700 1 Processor -- start 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.109+0000 7f3233b56700 1 -- start start 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f3233b56700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c1049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f3233b56700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c104fa0 con 0x7f322c1045b0 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32318f2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c1049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32318f2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c1049d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35068/0 (socket says 172.21.15.118:35068) 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32318f2700 1 -- 172.21.15.118:0/1210247887 learned_addr learned my addr 172.21.15.118:0/1210247887 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32318f2700 1 -- 172.21.15.118:0/1210247887 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c1050e0 con 0x7f322c1045b0 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32318f2700 1 --2- 172.21.15.118:0/1210247887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c1049d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f321c0098a0 tx=0x7f321c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dca3a021826799f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32308f0700 1 -- 172.21.15.118:0/1210247887 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f321c004180 con 0x7f322c1045b0 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32308f0700 1 -- 172.21.15.118:0/1210247887 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f321c0042e0 con 0x7f322c1045b0 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.110+0000 7f32308f0700 1 -- 172.21.15.118:0/1210247887 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f321c011b10 con 0x7f322c1045b0 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.111+0000 7f3233b56700 1 -- 172.21.15.118:0/1210247887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 msgr2=0x7f322c1049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:12.879 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.111+0000 7f3233b56700 1 --2- 172.21.15.118:0/1210247887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c1049d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f321c0098a0 tx=0x7f321c009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.111+0000 7f3233b56700 1 -- 172.21.15.118:0/1210247887 shutdown_connections 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.111+0000 7f3233b56700 1 --2- 172.21.15.118:0/1210247887 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c1049d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.111+0000 7f3233b56700 1 -- 172.21.15.118:0/1210247887 >> 172.21.15.118:0/1210247887 conn(0x7f322c0fff30 msgr2=0x7f322c102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.111+0000 7f3233b56700 1 -- 172.21.15.118:0/1210247887 shutdown_connections 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.111+0000 7f3233b56700 1 -- 172.21.15.118:0/1210247887 wait complete. 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3233b56700 1 Processor -- start 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3233b56700 1 -- start start 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3233b56700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c106e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3233b56700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c1073a0 con 0x7f322c1045b0 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f32318f2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c106e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f32318f2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c106e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35074/0 (socket says 172.21.15.118:35074) 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f32318f2700 1 -- 172.21.15.118:0/1700904269 learned_addr learned my addr 172.21.15.118:0/1700904269 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f32318f2700 1 -- 172.21.15.118:0/1700904269 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321c009550 con 0x7f322c1045b0 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f32318f2700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c106e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f321c009fb0 tx=0x7f321c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:12.880 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3222ffd700 1 -- 172.21.15.118:0/1700904269 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f321c0038b0 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3222ffd700 1 -- 172.21.15.118:0/1700904269 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f321c003a10 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3222ffd700 1 -- 172.21.15.118:0/1700904269 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f321c019770 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.112+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c1055e0 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.113+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c105a80 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.113+0000 7f3222ffd700 1 -- 172.21.15.118:0/1700904269 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f321c0198d0 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.113+0000 7f3222ffd700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f32180384b0 0x7f321803a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.113+0000 7f3222ffd700 1 -- 172.21.15.118:0/1700904269 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f321c04c570 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.113+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c0623e0 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.114+0000 7f32310f1700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f32180384b0 0x7f321803a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.114+0000 7f32310f1700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f32180384b0 0x7f321803a970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3228006fd0 tx=0x7f3228006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.118+0000 7f3222ffd700 1 -- 172.21.15.118:0/1700904269 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f321c019bd0 con 0x7f322c1045b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.343+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f322c0008d0 con 0x7f32180384b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.347+0000 7f3222ffd700 1 -- 172.21.15.118:0/1700904269 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f322c0008d0 con 0x7f32180384b0 2024-03-12T03:16:12.881 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.349+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f32180384b0 msgr2=0x7f321803a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.349+0000 7f3233b56700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f32180384b0 0x7f321803a970 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3228006fd0 tx=0x7f3228006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.349+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 msgr2=0x7f322c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.349+0000 7f3233b56700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c106e60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f321c009fb0 tx=0x7f321c011890 comp rx=0 tx=0).stop 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.350+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 shutdown_connections 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.350+0000 7f3233b56700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f32180384b0 0x7f321803a970 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.350+0000 7f3233b56700 1 --2- 172.21.15.118:0/1700904269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f322c1045b0 0x7f322c106e60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.350+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 >> 172.21.15.118:0/1700904269 conn(0x7f322c0fff30 msgr2=0x7f322c100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.350+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 shutdown_connections 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:12.350+0000 7f3233b56700 1 -- 172.21.15.118:0/1700904269 wait complete. 2024-03-12T03:16:12.882 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying ceph-exporter service with default placement... 2024-03-12T03:16:13.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:13 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:13.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:13 smithi118 ceph-mon[29160]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:13.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:13 smithi118 ceph-mon[29160]: Saving service crash spec with placement * 2024-03-12T03:16:13.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:13 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:14.637 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-03-12T03:16:14.637 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.321+0000 7f0cb7b3a700 1 Processor -- start 2024-03-12T03:16:14.637 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.321+0000 7f0cb7b3a700 1 -- start start 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.321+0000 7f0cb7b3a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb01067f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.321+0000 7f0cb7b3a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0106dc0 con 0x7f0cb01063d0 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.321+0000 7f0cb58d6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb01067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb58d6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb01067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35084/0 (socket says 172.21.15.118:35084) 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb58d6700 1 -- 172.21.15.118:0/4000899759 learned_addr learned my addr 172.21.15.118:0/4000899759 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb58d6700 1 -- 172.21.15.118:0/4000899759 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb01075d0 con 0x7f0cb01063d0 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb58d6700 1 --2- 172.21.15.118:0/4000899759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb01067f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0098a0 tx=0x7f0cac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b040ff48b56ea670 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb48d4700 1 -- 172.21.15.118:0/4000899759 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cac004180 con 0x7f0cb01063d0 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb48d4700 1 -- 172.21.15.118:0/4000899759 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cac0042e0 con 0x7f0cb01063d0 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb48d4700 1 -- 172.21.15.118:0/4000899759 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cac011b10 con 0x7f0cb01063d0 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/4000899759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 msgr2=0x7f0cb01067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.322+0000 7f0cb7b3a700 1 --2- 172.21.15.118:0/4000899759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb01067f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0cac0098a0 tx=0x7f0cac009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:14.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/4000899759 shutdown_connections 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 --2- 172.21.15.118:0/4000899759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb01067f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/4000899759 >> 172.21.15.118:0/4000899759 conn(0x7f0cb0101970 msgr2=0x7f0cb0103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/4000899759 shutdown_connections 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/4000899759 wait complete. 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 Processor -- start 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 -- start start 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb7b3a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb019a6c0 con 0x7f0cb01063d0 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.323+0000 7f0cb58d6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0cb58d6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb019a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35090/0 (socket says 172.21.15.118:35090) 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0cb58d6700 1 -- 172.21.15.118:0/2665387610 learned_addr learned my addr 172.21.15.118:0/2665387610 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0cb58d6700 1 -- 172.21.15.118:0/2665387610 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cac009550 con 0x7f0cb01063d0 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0cb58d6700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb019a180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009fb0 tx=0x7f0cac011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:14.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0ca6ffd700 1 -- 172.21.15.118:0/2665387610 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cac0038b0 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0ca6ffd700 1 -- 172.21.15.118:0/2665387610 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cac003a10 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0ca6ffd700 1 -- 172.21.15.118:0/2665387610 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cac019770 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb019a8c0 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.324+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb019ad60 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.325+0000 7f0ca6ffd700 1 -- 172.21.15.118:0/2665387610 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f0cac0198d0 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.325+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb010a800 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.325+0000 7f0ca6ffd700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0c9c038500 0x7f0c9c03a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.325+0000 7f0ca6ffd700 1 -- 172.21.15.118:0/2665387610 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0cac04c640 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.325+0000 7f0cb50d5700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0c9c038500 0x7f0c9c03a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.325+0000 7f0cb50d5700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0c9c038500 0x7f0c9c03a9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0006fd0 tx=0x7f0ca0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.330+0000 7f0ca6ffd700 1 -- 172.21.15.118:0/2665387610 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0cac019b80 con 0x7f0cb01063d0 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.572+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cb019da10 con 0x7f0c9c038500 2024-03-12T03:16:14.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.577+0000 7f0ca6ffd700 1 -- 172.21.15.118:0/2665387610 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f0cb019da10 con 0x7f0c9c038500 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0c9c038500 msgr2=0x7f0c9c03a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0c9c038500 0x7f0c9c03a9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0006fd0 tx=0x7f0ca0006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 msgr2=0x7f0cb019a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb019a180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009fb0 tx=0x7f0cac011890 comp rx=0 tx=0).stop 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 shutdown_connections 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0c9c038500 0x7f0c9c03a9c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 --2- 172.21.15.118:0/2665387610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0cb01063d0 0x7f0cb019a180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 >> 172.21.15.118:0/2665387610 conn(0x7f0cb0101970 msgr2=0x7f0cb0102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 shutdown_connections 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:13.579+0000 7f0cb7b3a700 1 -- 172.21.15.118:0/2665387610 wait complete. 2024-03-12T03:16:14.641 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying prometheus service with default placement... 2024-03-12T03:16:14.891 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:14 smithi118 ceph-mon[29160]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:14.891 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:14 smithi118 ceph-mon[29160]: Saving service ceph-exporter spec with placement * 2024-03-12T03:16:14.891 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:14 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:15.980 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:15 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:15.980 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:15 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:15.980 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:15 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:15.987 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-12T03:16:15.987 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.110+0000 7f7837bce700 1 Processor -- start 2024-03-12T03:16:15.987 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.110+0000 7f7837bce700 1 -- start start 2024-03-12T03:16:15.987 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.110+0000 7f7837bce700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783007a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:15.987 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.110+0000 7f7837bce700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783007a7f0 con 0x7f783007bdc0 2024-03-12T03:16:15.987 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.111+0000 7f783596a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783007a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.111+0000 7f783596a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783007a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40894/0 (socket says 172.21.15.118:40894) 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.111+0000 7f783596a700 1 -- 172.21.15.118:0/4253355618 learned_addr learned my addr 172.21.15.118:0/4253355618 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.111+0000 7f783596a700 1 -- 172.21.15.118:0/4253355618 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783007a930 con 0x7f783007bdc0 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.111+0000 7f783596a700 1 --2- 172.21.15.118:0/4253355618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783007a220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f782c0098a0 tx=0x7f782c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b1b5481722f61c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7834968700 1 -- 172.21.15.118:0/4253355618 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f782c004180 con 0x7f783007bdc0 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7834968700 1 -- 172.21.15.118:0/4253355618 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f782c0042e0 con 0x7f783007bdc0 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7834968700 1 -- 172.21.15.118:0/4253355618 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f782c011b10 con 0x7f783007bdc0 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7837bce700 1 -- 172.21.15.118:0/4253355618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 msgr2=0x7f783007a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7837bce700 1 --2- 172.21.15.118:0/4253355618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783007a220 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f782c0098a0 tx=0x7f782c009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7837bce700 1 -- 172.21.15.118:0/4253355618 shutdown_connections 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7837bce700 1 --2- 172.21.15.118:0/4253355618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783007a220 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.112+0000 7f7837bce700 1 -- 172.21.15.118:0/4253355618 >> 172.21.15.118:0/4253355618 conn(0x7f7830101880 msgr2=0x7f7830103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.113+0000 7f7837bce700 1 -- 172.21.15.118:0/4253355618 shutdown_connections 2024-03-12T03:16:15.988 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.113+0000 7f7837bce700 1 -- 172.21.15.118:0/4253355618 wait complete. 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.113+0000 7f7837bce700 1 Processor -- start 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.113+0000 7f7837bce700 1 -- start start 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f7837bce700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783019c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f7837bce700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783019c800 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f783596a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783019c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f783596a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783019c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40896/0 (socket says 172.21.15.118:40896) 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f783596a700 1 -- 172.21.15.118:0/734184234 learned_addr learned my addr 172.21.15.118:0/734184234 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f783596a700 1 -- 172.21.15.118:0/734184234 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c009550 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f783596a700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783019c2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f782c009fb0 tx=0x7f782c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f7826ffd700 1 -- 172.21.15.118:0/734184234 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f782c0038b0 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f783019ca00 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f7826ffd700 1 -- 172.21.15.118:0/734184234 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f782c003a10 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f7826ffd700 1 -- 172.21.15.118:0/734184234 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f782c019750 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.114+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f783019cea0 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.115+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f783010c700 con 0x7f783007bdc0 2024-03-12T03:16:15.989 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.116+0000 7f7826ffd700 1 -- 172.21.15.118:0/734184234 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f782c01a430 con 0x7f783007bdc0 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.116+0000 7f7826ffd700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f781c038500 0x7f781c03a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.116+0000 7f7826ffd700 1 -- 172.21.15.118:0/734184234 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f782c04c4c0 con 0x7f783007bdc0 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.116+0000 7f7835169700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f781c038500 0x7f781c03a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.116+0000 7f7835169700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f781c038500 0x7f781c03a9c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7820006fd0 tx=0x7f7820006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.121+0000 7f7826ffd700 1 -- 172.21.15.118:0/734184234 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f782c022310 con 0x7f783007bdc0 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.335+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f7830102ba0 con 0x7f781c038500 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.341+0000 7f7826ffd700 1 -- 172.21.15.118:0/734184234 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f7830102ba0 con 0x7f781c038500 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f781c038500 msgr2=0x7f781c03a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f781c038500 0x7f781c03a9c0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7820006fd0 tx=0x7f7820006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 msgr2=0x7f783019c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783019c2c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f782c009fb0 tx=0x7f782c011890 comp rx=0 tx=0).stop 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 shutdown_connections 2024-03-12T03:16:15.990 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f781c038500 0x7f781c03a9c0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:15.991 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 --2- 172.21.15.118:0/734184234 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f783007bdc0 0x7f783019c2c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:15.991 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.343+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 >> 172.21.15.118:0/734184234 conn(0x7f7830101880 msgr2=0x7f7830102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:15.991 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.344+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 shutdown_connections 2024-03-12T03:16:15.991 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:15.344+0000 7f7837bce700 1 -- 172.21.15.118:0/734184234 wait complete. 2024-03-12T03:16:15.991 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying grafana service with default placement... 2024-03-12T03:16:17.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:16 smithi118 ceph-mon[29160]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:17.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:16 smithi118 ceph-mon[29160]: Saving service prometheus spec with placement count:1 2024-03-12T03:16:17.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:16 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:18.043 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:17 smithi118 ceph-mon[29160]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:18.044 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:17 smithi118 ceph-mon[29160]: Saving service grafana spec with placement count:1 2024-03-12T03:16:18.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-12T03:16:18.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.418+0000 7f6dc1f63700 1 Processor -- start 2024-03-12T03:16:18.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.418+0000 7f6dc1f63700 1 -- start start 2024-03-12T03:16:18.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.418+0000 7f6dc1f63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc104a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:18.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.418+0000 7f6dc1f63700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc104fd0 con 0x7f6dbc1045e0 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dbb7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc104a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dbb7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc104a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40912/0 (socket says 172.21.15.118:40912) 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dbb7fe700 1 -- 172.21.15.118:0/2375774175 learned_addr learned my addr 172.21.15.118:0/2375774175 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dbb7fe700 1 -- 172.21.15.118:0/2375774175 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc107570 con 0x7f6dbc1045e0 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dbb7fe700 1 --2- 172.21.15.118:0/2375774175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc104a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6db00098a0 tx=0x7f6db0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2173b1105f0d043c server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dba7fc700 1 -- 172.21.15.118:0/2375774175 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db0004180 con 0x7f6dbc1045e0 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dba7fc700 1 -- 172.21.15.118:0/2375774175 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6db00042e0 con 0x7f6dbc1045e0 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.419+0000 7f6dba7fc700 1 -- 172.21.15.118:0/2375774175 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db0011b10 con 0x7f6dbc1045e0 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.420+0000 7f6dc1f63700 1 -- 172.21.15.118:0/2375774175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 msgr2=0x7f6dbc104a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.420+0000 7f6dc1f63700 1 --2- 172.21.15.118:0/2375774175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc104a00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6db00098a0 tx=0x7f6db0009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.420+0000 7f6dc1f63700 1 -- 172.21.15.118:0/2375774175 shutdown_connections 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.420+0000 7f6dc1f63700 1 --2- 172.21.15.118:0/2375774175 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc104a00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.420+0000 7f6dc1f63700 1 -- 172.21.15.118:0/2375774175 >> 172.21.15.118:0/2375774175 conn(0x7f6dbc0fff30 msgr2=0x7f6dbc102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:18.053 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.420+0000 7f6dc1f63700 1 -- 172.21.15.118:0/2375774175 shutdown_connections 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.420+0000 7f6dc1f63700 1 -- 172.21.15.118:0/2375774175 wait complete. 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.421+0000 7f6dc1f63700 1 Processor -- start 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.421+0000 7f6dc1f63700 1 -- start start 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.421+0000 7f6dc1f63700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc19a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.421+0000 7f6dc1f63700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc19a610 con 0x7f6dbc1045e0 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.421+0000 7f6dbb7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc19a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.421+0000 7f6dbb7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc19a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40918/0 (socket says 172.21.15.118:40918) 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.421+0000 7f6dbb7fe700 1 -- 172.21.15.118:0/3118798260 learned_addr learned my addr 172.21.15.118:0/3118798260 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.422+0000 7f6dbb7fe700 1 -- 172.21.15.118:0/3118798260 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db0009550 con 0x7f6dbc1045e0 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.422+0000 7f6dbb7fe700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc19a0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6db00117b0 tx=0x7f6db0011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.422+0000 7f6db8ff9700 1 -- 172.21.15.118:0/3118798260 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db00038b0 con 0x7f6dbc1045e0 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.422+0000 7f6db8ff9700 1 -- 172.21.15.118:0/3118798260 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6db0003a10 con 0x7f6dbc1045e0 2024-03-12T03:16:18.054 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.422+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc19a810 con 0x7f6dbc1045e0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.422+0000 7f6db8ff9700 1 -- 172.21.15.118:0/3118798260 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6db00246e0 con 0x7f6dbc1045e0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.422+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc19acb0 con 0x7f6dbc1045e0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.423+0000 7f6db8ff9700 1 -- 172.21.15.118:0/3118798260 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f6db002d3f0 con 0x7f6dbc1045e0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.423+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dbc10a7a0 con 0x7f6dbc1045e0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.423+0000 7f6db8ff9700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f6da40384f0 0x7f6da403a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.423+0000 7f6db8ff9700 1 -- 172.21.15.118:0/3118798260 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6db004c560 con 0x7f6dbc1045e0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.423+0000 7f6dbaffd700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f6da40384f0 0x7f6da403a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.424+0000 7f6dbaffd700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f6da40384f0 0x7f6da403a9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6dac006fd0 tx=0x7f6dac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.428+0000 7f6db8ff9700 1 -- 172.21.15.118:0/3118798260 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6db0010970 con 0x7f6dbc1045e0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.643+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dbc101260 con 0x7f6da40384f0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.648+0000 7f6db8ff9700 1 -- 172.21.15.118:0/3118798260 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f6dbc101260 con 0x7f6da40384f0 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.650+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f6da40384f0 msgr2=0x7f6da403a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.650+0000 7f6dc1f63700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f6da40384f0 0x7f6da403a9b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6dac006fd0 tx=0x7f6dac006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.650+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 msgr2=0x7f6dbc19a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:18.055 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.650+0000 7f6dc1f63700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc19a0d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6db00117b0 tx=0x7f6db0011890 comp rx=0 tx=0).stop 2024-03-12T03:16:18.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.650+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 shutdown_connections 2024-03-12T03:16:18.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.650+0000 7f6dc1f63700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f6da40384f0 0x7f6da403a9b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:18.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.650+0000 7f6dc1f63700 1 --2- 172.21.15.118:0/3118798260 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dbc1045e0 0x7f6dbc19a0d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:18.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.651+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 >> 172.21.15.118:0/3118798260 conn(0x7f6dbc0fff30 msgr2=0x7f6dbc100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:18.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.651+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 shutdown_connections 2024-03-12T03:16:18.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:16.651+0000 7f6dc1f63700 1 -- 172.21.15.118:0/3118798260 wait complete. 2024-03-12T03:16:18.056 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying node-exporter service with default placement... 2024-03-12T03:16:19.016 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:18 smithi118 ceph-mon[29160]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:19.317 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.508+0000 7f0fe275d700 1 Processor -- start 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.508+0000 7f0fe275d700 1 -- start start 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.508+0000 7f0fe275d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc106750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.508+0000 7f0fe275d700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc106d20 con 0x7f0fdc106330 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.508+0000 7f0fdbfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.508+0000 7f0fdbfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40934/0 (socket says 172.21.15.118:40934) 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.508+0000 7f0fdbfff700 1 -- 172.21.15.118:0/172426246 learned_addr learned my addr 172.21.15.118:0/172426246 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.509+0000 7f0fdbfff700 1 -- 172.21.15.118:0/172426246 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc107530 con 0x7f0fdc106330 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.509+0000 7f0fdbfff700 1 --2- 172.21.15.118:0/172426246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc106750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0098a0 tx=0x7f0fcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a660708171927c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.509+0000 7f0fdaffd700 1 -- 172.21.15.118:0/172426246 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fcc004180 con 0x7f0fdc106330 2024-03-12T03:16:19.318 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.509+0000 7f0fdaffd700 1 -- 172.21.15.118:0/172426246 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fcc0042e0 con 0x7f0fdc106330 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.509+0000 7f0fdaffd700 1 -- 172.21.15.118:0/172426246 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fcc011b10 con 0x7f0fdc106330 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 -- 172.21.15.118:0/172426246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 msgr2=0x7f0fdc106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 --2- 172.21.15.118:0/172426246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc106750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc0098a0 tx=0x7f0fcc009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 -- 172.21.15.118:0/172426246 shutdown_connections 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 --2- 172.21.15.118:0/172426246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc106750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 -- 172.21.15.118:0/172426246 >> 172.21.15.118:0/172426246 conn(0x7f0fdc1018b0 msgr2=0x7f0fdc103d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 -- 172.21.15.118:0/172426246 shutdown_connections 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 -- 172.21.15.118:0/172426246 wait complete. 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 Processor -- start 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.510+0000 7f0fe275d700 1 -- start start 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fe275d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc07ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fe275d700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc07bfa0 con 0x7f0fdc106330 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fdbfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc07ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fdbfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc07ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40948/0 (socket says 172.21.15.118:40948) 2024-03-12T03:16:19.319 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fdbfff700 1 -- 172.21.15.118:0/3404267420 learned_addr learned my addr 172.21.15.118:0/3404267420 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fdbfff700 1 -- 172.21.15.118:0/3404267420 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fcc009550 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fdbfff700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc07ba60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009fb0 tx=0x7f0fcc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fd97fa700 1 -- 172.21.15.118:0/3404267420 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fcc0038b0 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fd97fa700 1 -- 172.21.15.118:0/3404267420 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fcc003a10 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fdc07a1e0 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.511+0000 7f0fd97fa700 1 -- 172.21.15.118:0/3404267420 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fcc019770 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.512+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fdc07a680 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.512+0000 7f0fd97fa700 1 -- 172.21.15.118:0/3404267420 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f0fcc0198d0 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.512+0000 7f0fd97fa700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0fc40384b0 0x7f0fc403a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.513+0000 7f0fd97fa700 1 -- 172.21.15.118:0/3404267420 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0fcc01a8f0 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.513+0000 7f0fdb7fe700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0fc40384b0 0x7f0fc403a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.513+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc0623e0 con 0x7f0fdc106330 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.513+0000 7f0fdb7fe700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0fc40384b0 0x7f0fc403a970 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0006fd0 tx=0x7f0fd0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:19.320 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.517+0000 7f0fd97fa700 1 -- 172.21.15.118:0/3404267420 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0fcc019bd0 con 0x7f0fdc106330 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.724+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fdc10a710 con 0x7f0fc40384b0 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.728+0000 7f0fd97fa700 1 -- 172.21.15.118:0/3404267420 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f0fdc10a710 con 0x7f0fc40384b0 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0fc40384b0 msgr2=0x7f0fc403a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0fc40384b0 0x7f0fc403a970 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0006fd0 tx=0x7f0fd0006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 msgr2=0x7f0fdc07ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc07ba60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0fcc009fb0 tx=0x7f0fcc011890 comp rx=0 tx=0).stop 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 shutdown_connections 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f0fc40384b0 0x7f0fc403a970 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 --2- 172.21.15.118:0/3404267420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0fdc106330 0x7f0fdc07ba60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 >> 172.21.15.118:0/3404267420 conn(0x7f0fdc1018b0 msgr2=0x7f0fdc1024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 shutdown_connections 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:18.730+0000 7f0fe275d700 1 -- 172.21.15.118:0/3404267420 wait complete. 2024-03-12T03:16:19.321 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying alertmanager service with default placement... 2024-03-12T03:16:20.002 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:19 smithi118 ceph-mon[29160]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:20.002 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:19 smithi118 ceph-mon[29160]: Saving service node-exporter spec with placement * 2024-03-12T03:16:20.002 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:19 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.781+0000 7fa3344ac700 1 Processor -- start 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.781+0000 7fa3344ac700 1 -- start start 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.781+0000 7fa3344ac700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c105660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.781+0000 7fa3344ac700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c105c30 con 0x7fa32c105240 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa332248700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c105660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa332248700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c105660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40950/0 (socket says 172.21.15.118:40950) 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa332248700 1 -- 172.21.15.118:0/468065110 learned_addr learned my addr 172.21.15.118:0/468065110 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa332248700 1 -- 172.21.15.118:0/468065110 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c106440 con 0x7fa32c105240 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa332248700 1 --2- 172.21.15.118:0/468065110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c105660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3280098a0 tx=0x7fa328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61560b5daab1cd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:20.565 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa331246700 1 -- 172.21.15.118:0/468065110 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa328004180 con 0x7fa32c105240 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa331246700 1 -- 172.21.15.118:0/468065110 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3280042e0 con 0x7fa32c105240 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.782+0000 7fa331246700 1 -- 172.21.15.118:0/468065110 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa328011b10 con 0x7fa32c105240 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.783+0000 7fa3344ac700 1 -- 172.21.15.118:0/468065110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 msgr2=0x7fa32c105660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.783+0000 7fa3344ac700 1 --2- 172.21.15.118:0/468065110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c105660 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa3280098a0 tx=0x7fa328009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.783+0000 7fa3344ac700 1 -- 172.21.15.118:0/468065110 shutdown_connections 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.783+0000 7fa3344ac700 1 --2- 172.21.15.118:0/468065110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c105660 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.783+0000 7fa3344ac700 1 -- 172.21.15.118:0/468065110 >> 172.21.15.118:0/468065110 conn(0x7fa32c100840 msgr2=0x7fa32c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.783+0000 7fa3344ac700 1 -- 172.21.15.118:0/468065110 shutdown_connections 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.783+0000 7fa3344ac700 1 -- 172.21.15.118:0/468065110 wait complete. 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa3344ac700 1 Processor -- start 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa3344ac700 1 -- start start 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa3344ac700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa3344ac700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32c199550 con 0x7fa32c105240 2024-03-12T03:16:20.566 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa332248700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa332248700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40964/0 (socket says 172.21.15.118:40964) 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa332248700 1 -- 172.21.15.118:0/1013133530 learned_addr learned my addr 172.21.15.118:0/1013133530 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa332248700 1 -- 172.21.15.118:0/1013133530 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa328009550 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.784+0000 7fa332248700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c199010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3280117b0 tx=0x7fa328011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.785+0000 7fa3237fe700 1 -- 172.21.15.118:0/1013133530 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3280038b0 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.785+0000 7fa3237fe700 1 -- 172.21.15.118:0/1013133530 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa328003a10 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.785+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c199750 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.785+0000 7fa3237fe700 1 -- 172.21.15.118:0/1013133530 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3280246e0 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.785+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c199bf0 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.785+0000 7fa3237fe700 1 -- 172.21.15.118:0/1013133530 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7fa32802d3f0 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.786+0000 7fa3237fe700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fa318038540 0x7fa31803aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.786+0000 7fa3237fe700 1 -- 172.21.15.118:0/1013133530 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa32804c4e0 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.786+0000 7fa331a47700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fa318038540 0x7fa31803aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.786+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32c0623e0 con 0x7fa32c105240 2024-03-12T03:16:20.567 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.786+0000 7fa331a47700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fa318038540 0x7fa31803aa00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa31c006fd0 tx=0x7fa31c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.791+0000 7fa3237fe700 1 -- 172.21.15.118:0/1013133530 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa32802d6f0 con 0x7fa32c105240 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:19.999+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fa32c101b90 con 0x7fa318038540 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.004+0000 7fa3237fe700 1 -- 172.21.15.118:0/1013133530 <== mgr.14120 v2:172.21.15.118:6800/1066995478 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fa32c101b90 con 0x7fa318038540 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fa318038540 msgr2=0x7fa31803aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fa318038540 0x7fa31803aa00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa31c006fd0 tx=0x7fa31c006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 msgr2=0x7fa32c199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c199010 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa3280117b0 tx=0x7fa328011890 comp rx=0 tx=0).stop 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 shutdown_connections 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7fa318038540 0x7fa31803aa00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 --2- 172.21.15.118:0/1013133530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa32c105240 0x7fa32c199010 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 >> 172.21.15.118:0/1013133530 conn(0x7fa32c100840 msgr2=0x7fa32c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 shutdown_connections 2024-03-12T03:16:20.568 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:20.006+0000 7fa3344ac700 1 -- 172.21.15.118:0/1013133530 wait complete. 2024-03-12T03:16:20.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:20 smithi118 ceph-mon[29160]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:20.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:20 smithi118 ceph-mon[29160]: from='mgr.14120 172.21.15.118:0/1111884297' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:21.901 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:21 smithi118 ceph-mon[29160]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:21.901 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:21 smithi118 ceph-mon[29160]: Saving service alertmanager spec with placement count:1 2024-03-12T03:16:21.901 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:21 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2657235578' entity='client.admin' 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6362f1b700 1 Processor -- start 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6362f1b700 1 -- start start 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6362f1b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c07a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6362f1b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c07a7d0 con 0x7f635c07bda0 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6360cb7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c07a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6360cb7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c07a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40966/0 (socket says 172.21.15.118:40966) 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6360cb7700 1 -- 172.21.15.118:0/3305428771 learned_addr learned my addr 172.21.15.118:0/3305428771 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6360cb7700 1 -- 172.21.15.118:0/3305428771 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f635c07a910 con 0x7f635c07bda0 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f6360cb7700 1 --2- 172.21.15.118:0/3305428771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c07a200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63500098a0 tx=0x7f6350009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2e00fb19ce71ca server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f635b7fe700 1 -- 172.21.15.118:0/3305428771 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6350004180 con 0x7f635c07bda0 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f635b7fe700 1 -- 172.21.15.118:0/3305428771 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63500042e0 con 0x7f635c07bda0 2024-03-12T03:16:21.909 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.051+0000 7f635b7fe700 1 -- 172.21.15.118:0/3305428771 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6350011b10 con 0x7f635c07bda0 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.052+0000 7f6362f1b700 1 -- 172.21.15.118:0/3305428771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 msgr2=0x7f635c07a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.052+0000 7f6362f1b700 1 --2- 172.21.15.118:0/3305428771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c07a200 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f63500098a0 tx=0x7f6350009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.052+0000 7f6362f1b700 1 -- 172.21.15.118:0/3305428771 shutdown_connections 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.052+0000 7f6362f1b700 1 --2- 172.21.15.118:0/3305428771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c07a200 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.052+0000 7f6362f1b700 1 -- 172.21.15.118:0/3305428771 >> 172.21.15.118:0/3305428771 conn(0x7f635c101880 msgr2=0x7f635c103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.052+0000 7f6362f1b700 1 -- 172.21.15.118:0/3305428771 shutdown_connections 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.052+0000 7f6362f1b700 1 -- 172.21.15.118:0/3305428771 wait complete. 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6362f1b700 1 Processor -- start 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6362f1b700 1 -- start start 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6362f1b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6362f1b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635c19c7c0 con 0x7f635c07bda0 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6360cb7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6360cb7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40982/0 (socket says 172.21.15.118:40982) 2024-03-12T03:16:21.910 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6360cb7700 1 -- 172.21.15.118:0/2657235578 learned_addr learned my addr 172.21.15.118:0/2657235578 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6360cb7700 1 -- 172.21.15.118:0/2657235578 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6350009550 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.053+0000 7f6360cb7700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c19c280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f63500117b0 tx=0x7f6350011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.054+0000 7f6359ffb700 1 -- 172.21.15.118:0/2657235578 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63500038b0 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.054+0000 7f6359ffb700 1 -- 172.21.15.118:0/2657235578 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6350003a10 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.054+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635c19c9c0 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.054+0000 7f6359ffb700 1 -- 172.21.15.118:0/2657235578 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6350024780 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.054+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635c19ce60 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.055+0000 7f6359ffb700 1 -- 172.21.15.118:0/2657235578 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f63500248e0 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.055+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f635c10c6c0 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.055+0000 7f6359ffb700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f634c0384f0 0x7f634c03a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.055+0000 7f6359ffb700 1 -- 172.21.15.118:0/2657235578 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f635004c610 con 0x7f635c07bda0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.055+0000 7f635bfff700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f634c0384f0 0x7f634c03a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.055+0000 7f635bfff700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f634c0384f0 0x7f634c03a9b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6348006fd0 tx=0x7f6348006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:21.911 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.060+0000 7f6359ffb700 1 -- 172.21.15.118:0/2657235578 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6350024b90 con 0x7f635c07bda0 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.251+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f635c10c8b0 con 0x7f635c07bda0 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.254+0000 7f6359ffb700 1 -- 172.21.15.118:0/2657235578 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6350018350 con 0x7f635c07bda0 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f634c0384f0 msgr2=0x7f634c03a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f634c0384f0 0x7f634c03a9b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6348006fd0 tx=0x7f6348006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 msgr2=0x7f635c19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c19c280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f63500117b0 tx=0x7f6350011890 comp rx=0 tx=0).stop 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 shutdown_connections 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f634c0384f0 0x7f634c03a9b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 --2- 172.21.15.118:0/2657235578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f635c07bda0 0x7f635c19c280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 >> 172.21.15.118:0/2657235578 conn(0x7f635c101880 msgr2=0x7f635c102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 shutdown_connections 2024-03-12T03:16:21.912 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:21.256+0000 7f6362f1b700 1 -- 172.21.15.118:0/2657235578 wait complete. 2024-03-12T03:16:23.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:22 smithi118 ceph-mon[29160]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:23.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:22 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/434413029' entity='client.admin' 2024-03-12T03:16:23.164 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.352+0000 7f8fa44ba700 1 Processor -- start 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.352+0000 7f8fa44ba700 1 -- start start 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.352+0000 7f8fa44ba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c109220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa44ba700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c079d70 con 0x7f8f9c106e30 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa2256700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa2256700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40984/0 (socket says 172.21.15.118:40984) 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa2256700 1 -- 172.21.15.118:0/1900298483 learned_addr learned my addr 172.21.15.118:0/1900298483 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa2256700 1 -- 172.21.15.118:0/1900298483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c109760 con 0x7f8f9c106e30 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa2256700 1 --2- 172.21.15.118:0/1900298483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c109220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0098a0 tx=0x7f8f8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539b175687c75edc server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa1254700 1 -- 172.21.15.118:0/1900298483 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8c004180 con 0x7f8f9c106e30 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa1254700 1 -- 172.21.15.118:0/1900298483 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f8c0042e0 con 0x7f8f9c106e30 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.353+0000 7f8fa1254700 1 -- 172.21.15.118:0/1900298483 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8c011b10 con 0x7f8f9c106e30 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.354+0000 7f8fa44ba700 1 -- 172.21.15.118:0/1900298483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 msgr2=0x7f8f9c109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.354+0000 7f8fa44ba700 1 --2- 172.21.15.118:0/1900298483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c109220 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0098a0 tx=0x7f8f8c009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:23.165 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.354+0000 7f8fa44ba700 1 -- 172.21.15.118:0/1900298483 shutdown_connections 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.354+0000 7f8fa44ba700 1 --2- 172.21.15.118:0/1900298483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c109220 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.354+0000 7f8fa44ba700 1 -- 172.21.15.118:0/1900298483 >> 172.21.15.118:0/1900298483 conn(0x7f8f9c100840 msgr2=0x7f8f9c102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.354+0000 7f8fa44ba700 1 -- 172.21.15.118:0/1900298483 shutdown_connections 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.354+0000 7f8fa44ba700 1 -- 172.21.15.118:0/1900298483 wait complete. 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa44ba700 1 Processor -- start 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa44ba700 1 -- start start 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa44ba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa44ba700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c19a630 con 0x7f8f9c106e30 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa2256700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa2256700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c19a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40988/0 (socket says 172.21.15.118:40988) 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa2256700 1 -- 172.21.15.118:0/434413029 learned_addr learned my addr 172.21.15.118:0/434413029 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa2256700 1 -- 172.21.15.118:0/434413029 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c009550 con 0x7f8f9c106e30 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8fa2256700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c19a0f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c011670 tx=0x7f8f8c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:23.166 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8f9b7fe700 1 -- 172.21.15.118:0/434413029 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8c0038b0 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.355+0000 7f8f9b7fe700 1 -- 172.21.15.118:0/434413029 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f8c003a10 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.356+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c19a830 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.356+0000 7f8f9b7fe700 1 -- 172.21.15.118:0/434413029 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f8c01a7c0 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.356+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c19acd0 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.356+0000 7f8f9b7fe700 1 -- 172.21.15.118:0/434413029 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f8f8c01a9a0 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.356+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c10a4a0 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.357+0000 7f8f9b7fe700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f8f84038500 0x7f8f8403a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.357+0000 7f8f9b7fe700 1 -- 172.21.15.118:0/434413029 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8f8c04c710 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.357+0000 7f8fa1a55700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f8f84038500 0x7f8f8403a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.357+0000 7f8fa1a55700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f8f84038500 0x7f8f8403a9c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f90006fd0 tx=0x7f8f90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.361+0000 7f8f9b7fe700 1 -- 172.21.15.118:0/434413029 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8f8c01ac50 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.550+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f8f9c10a690 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.554+0000 7f8f9b7fe700 1 -- 172.21.15.118:0/434413029 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f8f8c024c10 con 0x7f8f9c106e30 2024-03-12T03:16:23.167 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f8f84038500 msgr2=0x7f8f8403a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f8f84038500 0x7f8f8403a9c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8f90006fd0 tx=0x7f8f90006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 msgr2=0x7f8f9c19a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c19a0f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c011670 tx=0x7f8f8c011890 comp rx=0 tx=0).stop 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 shutdown_connections 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f8f84038500 0x7f8f8403a9c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 --2- 172.21.15.118:0/434413029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8f9c106e30 0x7f8f9c19a0f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 >> 172.21.15.118:0/434413029 conn(0x7f8f9c100840 msgr2=0x7f8f9c101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 shutdown_connections 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:22.556+0000 7f8fa44ba700 1 -- 172.21.15.118:0/434413029 wait complete. 2024-03-12T03:16:23.168 INFO:teuthology.orchestra.run.smithi118.stdout:Enabling the dashboard module... 2024-03-12T03:16:25.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:24 smithi118 ceph-mon[29160]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:25.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:24 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3531589482' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T03:16:25.441 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f9634700 1 Processor -- start 2024-03-12T03:16:25.447 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f9634700 1 -- start start 2024-03-12T03:16:25.447 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f9634700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f4106bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:25.447 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f9634700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f41071c0 con 0x7f49f41067d0 2024-03-12T03:16:25.447 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f2ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f4106bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f2ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f4106bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40992/0 (socket says 172.21.15.118:40992) 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f2ffd700 1 -- 172.21.15.118:0/73591917 learned_addr learned my addr 172.21.15.118:0/73591917 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.595+0000 7f49f2ffd700 1 -- 172.21.15.118:0/73591917 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f4107300 con 0x7f49f41067d0 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f2ffd700 1 --2- 172.21.15.118:0/73591917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f4106bf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f49e40098a0 tx=0x7f49e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34a36216d29e158 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f1ffb700 1 -- 172.21.15.118:0/73591917 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e4004180 con 0x7f49f41067d0 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f1ffb700 1 -- 172.21.15.118:0/73591917 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49e40042e0 con 0x7f49f41067d0 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f1ffb700 1 -- 172.21.15.118:0/73591917 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e4011b10 con 0x7f49f41067d0 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f9634700 1 -- 172.21.15.118:0/73591917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 msgr2=0x7f49f4106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f9634700 1 --2- 172.21.15.118:0/73591917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f4106bf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f49e40098a0 tx=0x7f49e4009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f9634700 1 -- 172.21.15.118:0/73591917 shutdown_connections 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f9634700 1 --2- 172.21.15.118:0/73591917 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f4106bf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f9634700 1 -- 172.21.15.118:0/73591917 >> 172.21.15.118:0/73591917 conn(0x7f49f40fff20 msgr2=0x7f49f4102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f9634700 1 -- 172.21.15.118:0/73591917 shutdown_connections 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.596+0000 7f49f9634700 1 -- 172.21.15.118:0/73591917 wait complete. 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.597+0000 7f49f9634700 1 Processor -- start 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.597+0000 7f49f9634700 1 -- start start 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.597+0000 7f49f9634700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.597+0000 7f49f9634700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f419c830 con 0x7f49f41067d0 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.597+0000 7f49f2ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.597+0000 7f49f2ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f419c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:41000/0 (socket says 172.21.15.118:41000) 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.597+0000 7f49f2ffd700 1 -- 172.21.15.118:0/3531589482 learned_addr learned my addr 172.21.15.118:0/3531589482 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.598+0000 7f49f2ffd700 1 -- 172.21.15.118:0/3531589482 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e4009550 con 0x7f49f41067d0 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.598+0000 7f49f2ffd700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f419c2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49e40117b0 tx=0x7f49e4011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.598+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e40038b0 con 0x7f49f41067d0 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.598+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49e4003a10 con 0x7f49f41067d0 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.598+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f419ca30 con 0x7f49f41067d0 2024-03-12T03:16:25.449 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.598+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49e4024780 con 0x7f49f41067d0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.598+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f419ced0 con 0x7f49f41067d0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.599+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 45389+0+0 (secure 0 0 0) 0x7f49e4024900 con 0x7f49f41067d0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.599+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f4104f60 con 0x7f49f41067d0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.599+0000 7f49dbfff700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f49dc0384f0 0x7f49dc03a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.599+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f49e404c4b0 con 0x7f49f41067d0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.599+0000 7f49f27fc700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f49dc0384f0 0x7f49dc03a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.599+0000 7f49f27fc700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f49dc0384f0 0x7f49dc03a9b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f49e8006fd0 tx=0x7f49e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.604+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f49e4024bb0 con 0x7f49f41067d0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:23.853+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f49f4195a30 con 0x7f49f41067d0 2024-03-12T03:16:25.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.808+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 9) v1 ==== 45402+0+0 (secure 0 0 0) 0x7f49e401ac10 con 0x7f49f41067d0 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.810+0000 7f49dbfff700 1 -- 172.21.15.118:0/3531589482 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f49e4018b40 con 0x7f49f41067d0 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f49dc0384f0 msgr2=0x7f49dc03a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f49dc0384f0 0x7f49dc03a9b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f49e8006fd0 tx=0x7f49e8006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 msgr2=0x7f49f419c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f419c2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49e40117b0 tx=0x7f49e4011890 comp rx=0 tx=0).stop 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 shutdown_connections 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f49dc0384f0 0x7f49dc03a9b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 --2- 172.21.15.118:0/3531589482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49f41067d0 0x7f49f419c2f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 >> 172.21.15.118:0/3531589482 conn(0x7f49f40fff20 msgr2=0x7f49f4192b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 shutdown_connections 2024-03-12T03:16:25.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:24.813+0000 7f49f9634700 1 -- 172.21.15.118:0/3531589482 wait complete. 2024-03-12T03:16:26.054 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:25 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3531589482' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T03:16:26.054 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:25 smithi118 ceph-mon[29160]: mgrmap e9: smithi118.zzzuvl(active, since 27s) 2024-03-12T03:16:26.770 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:16:26.770 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "active_name": "smithi118.zzzuvl", 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.846+0000 7f02f9b07700 1 Processor -- start 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.846+0000 7f02f9b07700 1 -- start start 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.846+0000 7f02f9b07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f4104d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.846+0000 7f02f9b07700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f4105360 con 0x7f02f4104970 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f4104d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f4104d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60444/0 (socket says 172.21.15.118:60444) 2024-03-12T03:16:26.771 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f37fe700 1 -- 172.21.15.118:0/609560273 learned_addr learned my addr 172.21.15.118:0/609560273 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f37fe700 1 -- 172.21.15.118:0/609560273 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f4105b70 con 0x7f02f4104970 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f37fe700 1 --2- 172.21.15.118:0/609560273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f4104d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f02e80098a0 tx=0x7f02e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4c5b9196efa2dd server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f27fc700 1 -- 172.21.15.118:0/609560273 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e8004180 con 0x7f02f4104970 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f27fc700 1 -- 172.21.15.118:0/609560273 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02e80042e0 con 0x7f02f4104970 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.847+0000 7f02f27fc700 1 -- 172.21.15.118:0/609560273 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e8011b10 con 0x7f02f4104970 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 -- 172.21.15.118:0/609560273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 msgr2=0x7f02f4104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 --2- 172.21.15.118:0/609560273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f4104d90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f02e80098a0 tx=0x7f02e8009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 -- 172.21.15.118:0/609560273 shutdown_connections 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 --2- 172.21.15.118:0/609560273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f4104d90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 -- 172.21.15.118:0/609560273 >> 172.21.15.118:0/609560273 conn(0x7f02f40fff30 msgr2=0x7f02f4102350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:26.772 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 -- 172.21.15.118:0/609560273 shutdown_connections 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 -- 172.21.15.118:0/609560273 wait complete. 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.848+0000 7f02f9b07700 1 Processor -- start 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f9b07700 1 -- start start 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f9b07700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f418fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f9b07700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f418ff70 con 0x7f02f4104970 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f418fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f418fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60448/0 (socket says 172.21.15.118:60448) 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f37fe700 1 -- 172.21.15.118:0/3601350510 learned_addr learned my addr 172.21.15.118:0/3601350510 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f37fe700 1 -- 172.21.15.118:0/3601350510 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8009550 con 0x7f02f4104970 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.849+0000 7f02f37fe700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f418fa30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f02e80117b0 tx=0x7f02e8011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.850+0000 7f02f0ff9700 1 -- 172.21.15.118:0/3601350510 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e80038b0 con 0x7f02f4104970 2024-03-12T03:16:26.773 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.850+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f418e1b0 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.850+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f418e650 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.850+0000 7f02f0ff9700 1 -- 172.21.15.118:0/3601350510 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02e8003a10 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.850+0000 7f02f0ff9700 1 -- 172.21.15.118:0/3601350510 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e80246e0 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.850+0000 7f02f0ff9700 1 -- 172.21.15.118:0/3601350510 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 9) v1 ==== 45402+0+0 (secure 0 0 0) 0x7f02e802d3f0 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.851+0000 7f02f0ff9700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 0x7f02dc03aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.851+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e00052f0 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.851+0000 7f02f2ffd700 1 -- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 msgr2=0x7f02dc03aa50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1066995478 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.851+0000 7f02f0ff9700 1 -- 172.21.15.118:0/3601350510 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f02e804d570 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.851+0000 7f02f2ffd700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 0x7f02dc03aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:25.856+0000 7f02f0ff9700 1 -- 172.21.15.118:0/3601350510 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f02e8010970 con 0x7f02f4104970 2024-03-12T03:16:26.774 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.052+0000 7f02f2ffd700 1 -- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 msgr2=0x7f02dc03aa50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1066995478 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.052+0000 7f02f2ffd700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 0x7f02dc03aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.113+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f02e00061d0 con 0x7f02f4104970 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.113+0000 7f02f0ff9700 1 -- 172.21.15.118:0/3601350510 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f02e800fb60 con 0x7f02f4104970 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 msgr2=0x7f02dc03aa50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 0x7f02dc03aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 msgr2=0x7f02f418fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f418fa30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f02e80117b0 tx=0x7f02e8011890 comp rx=0 tx=0).stop 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 shutdown_connections 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f02dc038590 0x7f02dc03aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 --2- 172.21.15.118:0/3601350510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f02f4104970 0x7f02f418fa30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 >> 172.21.15.118:0/3601350510 conn(0x7f02f40fff30 msgr2=0x7f02f4101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:26.775 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.115+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 shutdown_connections 2024-03-12T03:16:26.776 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:26.116+0000 7f02f9b07700 1 -- 172.21.15.118:0/3601350510 wait complete. 2024-03-12T03:16:26.776 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for the mgr to restart... 2024-03-12T03:16:26.776 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr epoch 9... 2024-03-12T03:16:27.025 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:26 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3601350510' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:16:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: Active manager daemon smithi118.zzzuvl restarted 2024-03-12T03:16:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: Activating manager daemon smithi118.zzzuvl 2024-03-12T03:16:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T03:16:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: mgrmap e10: smithi118.zzzuvl(active, starting, since 0.00351653s) 2024-03-12T03:16:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:16:32.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr metadata", "who": "smithi118.zzzuvl", "id": "smithi118.zzzuvl"}]: dispatch 2024-03-12T03:16:32.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:16:32.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:16:32.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:16:32.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: Manager daemon smithi118.zzzuvl is now available 2024-03-12T03:16:32.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:32.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:31 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:16:32.802 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:32 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:16:32.802 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:32 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/trash_purge_schedule"}]: dispatch 2024-03-12T03:16:32.802 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:32 smithi118 ceph-mon[29160]: mgrmap e11: smithi118.zzzuvl(active, since 1.00559s) 2024-03-12T03:16:33.371 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-12T03:16:33.371 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-12T03:16:33.371 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-12T03:16:33.371 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-12T03:16:33.371 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eef15700 1 Processor -- start 2024-03-12T03:16:33.371 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eef15700 1 -- start start 2024-03-12T03:16:33.371 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eef15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e807a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eef15700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e807a850 con 0x7f54e807be20 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eccb1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e807a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eccb1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e807a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60458/0 (socket says 172.21.15.118:60458) 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eccb1700 1 -- 172.21.15.118:0/568638912 learned_addr learned my addr 172.21.15.118:0/568638912 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eccb1700 1 -- 172.21.15.118:0/568638912 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e807a990 con 0x7f54e807be20 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54eccb1700 1 --2- 172.21.15.118:0/568638912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e807a280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0098a0 tx=0x7f54dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bee5507c8686d16 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54e77fe700 1 -- 172.21.15.118:0/568638912 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54dc004180 con 0x7f54e807be20 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54e77fe700 1 -- 172.21.15.118:0/568638912 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54dc0042e0 con 0x7f54e807be20 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.230+0000 7f54e77fe700 1 -- 172.21.15.118:0/568638912 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54dc011b10 con 0x7f54e807be20 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.231+0000 7f54eef15700 1 -- 172.21.15.118:0/568638912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 msgr2=0x7f54e807a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.231+0000 7f54eef15700 1 --2- 172.21.15.118:0/568638912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e807a280 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0098a0 tx=0x7f54dc009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.231+0000 7f54eef15700 1 -- 172.21.15.118:0/568638912 shutdown_connections 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.231+0000 7f54eef15700 1 --2- 172.21.15.118:0/568638912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e807a280 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:33.372 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.231+0000 7f54eef15700 1 -- 172.21.15.118:0/568638912 >> 172.21.15.118:0/568638912 conn(0x7f54e8101970 msgr2=0x7f54e8103db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.231+0000 7f54eef15700 1 -- 172.21.15.118:0/568638912 shutdown_connections 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.231+0000 7f54eef15700 1 -- 172.21.15.118:0/568638912 wait complete. 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eef15700 1 Processor -- start 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eef15700 1 -- start start 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eef15700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eef15700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e819c8c0 con 0x7f54e807be20 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eccb1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eccb1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e819c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60462/0 (socket says 172.21.15.118:60462) 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eccb1700 1 -- 172.21.15.118:0/1676671943 learned_addr learned my addr 172.21.15.118:0/1676671943 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:33.373 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.232+0000 7f54eccb1700 1 -- 172.21.15.118:0/1676671943 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc009550 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.233+0000 7f54eccb1700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e819c380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009fb0 tx=0x7f54dc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.233+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54dc0038b0 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.233+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e819cac0 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.233+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54dc003a10 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.233+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54dc019770 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.233+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e819cf60 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.234+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 9) v1 ==== 45402+0+0 (secure 0 0 0) 0x7f54dc0198d0 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.234+0000 7f54e5ffb700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.234+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 --> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54e810c7a0 con 0x7f54d8038500 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.234+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f54dc04d620 con 0x7f54e807be20 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.234+0000 7f54e7fff700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 msgr2=0x7f54d803a9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1066995478 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.234+0000 7f54e7fff700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.435+0000 7f54e7fff700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 msgr2=0x7f54d803a9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1066995478 2024-03-12T03:16:33.374 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.435+0000 7f54e7fff700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.836+0000 7f54e7fff700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 msgr2=0x7f54d803a9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1066995478 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:27.836+0000 7f54e7fff700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:28.637+0000 7f54e7fff700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 msgr2=0x7f54d803a9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1066995478 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:28.637+0000 7f54e7fff700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:30.238+0000 7f54e7fff700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 msgr2=0x7f54d803a9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1066995478 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:30.238+0000 7f54e7fff700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:31.649+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mgrmap(e 10) v1 ==== 45169+0+0 (secure 0 0 0) 0x7f54dc01ac10 con 0x7f54e807be20 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:31.649+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 msgr2=0x7f54d803a9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:31.649+0000 7f54e5ffb700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.651+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 11) v1 ==== 45296+0+0 (secure 0 0 0) 0x7f54dc04cc10 con 0x7f54e807be20 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.651+0000 7f54e5ffb700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f54d803be10 0x7f54d803e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.651+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f54e810c7a0 con 0x7f54d803be10 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.652+0000 7f54e7fff700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f54d803be10 0x7f54d803e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:33.375 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.653+0000 7f54e7fff700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f54d803be10 0x7f54d803e200 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f54d4003a10 tx=0x7f54d40092b0 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.653+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mgr.14164 v2:172.21.15.118:6800/180481365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f54e810c7a0 con 0x7f54d803be10 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.659+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f54e810c910 con 0x7f54d803be10 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.659+0000 7f54e5ffb700 1 -- 172.21.15.118:0/1676671943 <== mgr.14164 v2:172.21.15.118:6800/180481365 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f54e810c910 con 0x7f54d803be10 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f54d803be10 msgr2=0x7f54d803e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f54d803be10 0x7f54d803e200 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f54d4003a10 tx=0x7f54d40092b0 comp rx=0 tx=0).stop 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 msgr2=0x7f54e819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e819c380 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f54dc009fb0 tx=0x7f54dc011890 comp rx=0 tx=0).stop 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 shutdown_connections 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f54d803be10 0x7f54d803e200 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e807be20 0x7f54e819c380 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 --2- 172.21.15.118:0/1676671943 >> [v2:172.21.15.118:6800/1066995478,v1:172.21.15.118:6801/1066995478] conn(0x7f54d8038500 0x7f54d803a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 >> 172.21.15.118:0/1676671943 conn(0x7f54e8101970 msgr2=0x7f54e8102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:33.376 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 shutdown_connections 2024-03-12T03:16:33.377 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:32.660+0000 7f54eef15700 1 -- 172.21.15.118:0/1676671943 wait complete. 2024-03-12T03:16:33.377 INFO:teuthology.orchestra.run.smithi118.stdout:mgr epoch 9 is available 2024-03-12T03:16:33.377 INFO:teuthology.orchestra.run.smithi118.stdout:Generating a dashboard self-signed certificate... 2024-03-12T03:16:34.038 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:33 smithi118 ceph-mon[29160]: [12/Mar/2024:03:16:32] ENGINE Bus STARTING 2024-03-12T03:16:34.038 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:33 smithi118 ceph-mon[29160]: [12/Mar/2024:03:16:32] ENGINE Serving on http://172.21.15.118:8765 2024-03-12T03:16:34.038 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:33 smithi118 ceph-mon[29160]: [12/Mar/2024:03:16:32] ENGINE Serving on https://172.21.15.118:7150 2024-03-12T03:16:34.038 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:33 smithi118 ceph-mon[29160]: [12/Mar/2024:03:16:32] ENGINE Bus STARTED 2024-03-12T03:16:34.038 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:33 smithi118 ceph-mon[29160]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:16:34.038 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:33 smithi118 ceph-mon[29160]: from='client.14168 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:16:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:34 smithi118 ceph-mon[29160]: mgrmap e12: smithi118.zzzuvl(active, since 2s) 2024-03-12T03:16:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:34 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:34 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:34 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.814+0000 7fc0933a7700 1 Processor -- start 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.814+0000 7fc0933a7700 1 -- start start 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc0933a7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c106c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc0933a7700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c107200 con 0x7fc08c106810 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc091143700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c106c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc091143700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c106c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60516/0 (socket says 172.21.15.118:60516) 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc091143700 1 -- 172.21.15.118:0/1957853413 learned_addr learned my addr 172.21.15.118:0/1957853413 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:34.818 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc091143700 1 -- 172.21.15.118:0/1957853413 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c107340 con 0x7fc08c106810 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc091143700 1 --2- 172.21.15.118:0/1957853413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c106c30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc084009e80 tx=0x7fc08400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d17f34d0a6f1787 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc08bfff700 1 -- 172.21.15.118:0/1957853413 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08400e070 con 0x7fc08c106810 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc08bfff700 1 -- 172.21.15.118:0/1957853413 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc08400b8c0 con 0x7fc08c106810 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.815+0000 7fc08bfff700 1 -- 172.21.15.118:0/1957853413 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0840044d0 con 0x7fc08c106810 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.816+0000 7fc0933a7700 1 -- 172.21.15.118:0/1957853413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 msgr2=0x7fc08c106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.816+0000 7fc0933a7700 1 --2- 172.21.15.118:0/1957853413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c106c30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc084009e80 tx=0x7fc08400b330 comp rx=0 tx=0).stop 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.816+0000 7fc0933a7700 1 -- 172.21.15.118:0/1957853413 shutdown_connections 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.816+0000 7fc0933a7700 1 --2- 172.21.15.118:0/1957853413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c106c30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.816+0000 7fc0933a7700 1 -- 172.21.15.118:0/1957853413 >> 172.21.15.118:0/1957853413 conn(0x7fc08c0fffc0 msgr2=0x7fc08c1023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.816+0000 7fc0933a7700 1 -- 172.21.15.118:0/1957853413 shutdown_connections 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.816+0000 7fc0933a7700 1 -- 172.21.15.118:0/1957853413 wait complete. 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc0933a7700 1 Processor -- start 2024-03-12T03:16:34.819 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc0933a7700 1 -- start start 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc0933a7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc0933a7700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c19c7d0 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc091143700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc091143700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c19c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60522/0 (socket says 172.21.15.118:60522) 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc091143700 1 -- 172.21.15.118:0/3594231555 learned_addr learned my addr 172.21.15.118:0/3594231555 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc091143700 1 -- 172.21.15.118:0/3594231555 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084009b30 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc091143700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c19c290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0840072a0 tx=0x7fc08400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc08a7fc700 1 -- 172.21.15.118:0/3594231555 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08400e040 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c19c9d0 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.817+0000 7fc08a7fc700 1 -- 172.21.15.118:0/3594231555 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc084003dc0 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.818+0000 7fc08a7fc700 1 -- 172.21.15.118:0/3594231555 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc084012750 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.818+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c19ced0 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.818+0000 7fc08a7fc700 1 -- 172.21.15.118:0/3594231555 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 12) v1 ==== 45402+0+0 (secure 0 0 0) 0x7fc084019040 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.818+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c104fa0 con 0x7fc08c106810 2024-03-12T03:16:34.820 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.818+0000 7fc08a7fc700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc07c038540 0x7fc07c03aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.819+0000 7fc08a7fc700 1 -- 172.21.15.118:0/3594231555 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc08404bf90 con 0x7fc08c106810 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.819+0000 7fc090942700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc07c038540 0x7fc07c03aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.819+0000 7fc090942700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc07c038540 0x7fc07c03aa00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc078006fd0 tx=0x7fc078006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:33.823+0000 7fc08a7fc700 1 -- 172.21.15.118:0/3594231555 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc08407f960 con 0x7fc08c106810 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.035+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fc08c194230 con 0x7fc07c038540 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.096+0000 7fc08a7fc700 1 -- 172.21.15.118:0/3594231555 <== mgr.14164 v2:172.21.15.118:6800/180481365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fc08c194230 con 0x7fc07c038540 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc07c038540 msgr2=0x7fc07c03aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc07c038540 0x7fc07c03aa00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc078006fd0 tx=0x7fc078006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 msgr2=0x7fc08c19c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c19c290 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc0840072a0 tx=0x7fc08400be00 comp rx=0 tx=0).stop 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 shutdown_connections 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc07c038540 0x7fc07c03aa00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 --2- 172.21.15.118:0/3594231555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc08c106810 0x7fc08c19c290 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 >> 172.21.15.118:0/3594231555 conn(0x7fc08c0fffc0 msgr2=0x7fc08c192a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:34.822 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 shutdown_connections 2024-03-12T03:16:34.822 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:34.099+0000 7fc0933a7700 1 -- 172.21.15.118:0/3594231555 wait complete. 2024-03-12T03:16:34.822 INFO:teuthology.orchestra.run.smithi118.stdout:Creating initial admin user... 2024-03-12T03:16:35.767 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:35 smithi118 ceph-mon[29160]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:35.767 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:35 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:37.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:36 smithi118 ceph-mon[29160]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:37.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:36 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:37.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8.L02lLz1UPGKTN/1sHVvO3LtC6R3.2HQ1uglFLBGTW09vhvXsQWW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710213395, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T03:16:37.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.273+0000 7f160ee54700 1 Processor -- start 2024-03-12T03:16:37.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.273+0000 7f160ee54700 1 -- start start 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.273+0000 7f160ee54700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f1608079130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.273+0000 7f160ee54700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1608079700 con 0x7f160807acd0 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.273+0000 7f160cbf0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f1608079130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.274+0000 7f160cbf0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f1608079130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39954/0 (socket says 172.21.15.118:39954) 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.274+0000 7f160cbf0700 1 -- 172.21.15.118:0/4107691807 learned_addr learned my addr 172.21.15.118:0/4107691807 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.274+0000 7f160cbf0700 1 -- 172.21.15.118:0/4107691807 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1608079840 con 0x7f160807acd0 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.274+0000 7f160cbf0700 1 --2- 172.21.15.118:0/4107691807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f1608079130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0098a0 tx=0x7f15fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc4f651daff40d server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.274+0000 7f16077fe700 1 -- 172.21.15.118:0/4107691807 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15fc004180 con 0x7f160807acd0 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.274+0000 7f16077fe700 1 -- 172.21.15.118:0/4107691807 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15fc0042e0 con 0x7f160807acd0 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.274+0000 7f16077fe700 1 -- 172.21.15.118:0/4107691807 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15fc011b10 con 0x7f160807acd0 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 -- 172.21.15.118:0/4107691807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 msgr2=0x7f1608079130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:37.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 --2- 172.21.15.118:0/4107691807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f1608079130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0098a0 tx=0x7f15fc009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 -- 172.21.15.118:0/4107691807 shutdown_connections 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 --2- 172.21.15.118:0/4107691807 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f1608079130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 -- 172.21.15.118:0/4107691807 >> 172.21.15.118:0/4107691807 conn(0x7f1608101900 msgr2=0x7f1608103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 -- 172.21.15.118:0/4107691807 shutdown_connections 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 -- 172.21.15.118:0/4107691807 wait complete. 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.275+0000 7f160ee54700 1 Processor -- start 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160ee54700 1 -- start start 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160ee54700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f160819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160ee54700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160819c850 con 0x7f160807acd0 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160cbf0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f160819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160cbf0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f160819c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39956/0 (socket says 172.21.15.118:39956) 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160cbf0700 1 -- 172.21.15.118:0/4234952457 learned_addr learned my addr 172.21.15.118:0/4234952457 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160cbf0700 1 -- 172.21.15.118:0/4234952457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc009550 con 0x7f160807acd0 2024-03-12T03:16:37.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160cbf0700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f160819c310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0117b0 tx=0x7f15fc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f1605ffb700 1 -- 172.21.15.118:0/4234952457 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15fc0038b0 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f1605ffb700 1 -- 172.21.15.118:0/4234952457 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15fc003a10 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160819ca50 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.276+0000 7f1605ffb700 1 -- 172.21.15.118:0/4234952457 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15fc0246e0 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.277+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160819cef0 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.277+0000 7f1605ffb700 1 -- 172.21.15.118:0/4234952457 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 12) v1 ==== 45402+0+0 (secure 0 0 0) 0x7f15fc02d3f0 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.278+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160810c6e0 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.278+0000 7f1605ffb700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f15f8038590 0x7f15f803aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.278+0000 7f1607fff700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f15f8038590 0x7f15f803aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.280+0000 7f1605ffb700 1 -- 172.21.15.118:0/4234952457 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f15fc04c680 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.280+0000 7f1607fff700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f15f8038590 0x7f15f803aa50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15f4006fd0 tx=0x7f15f4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.287+0000 7f1605ffb700 1 -- 172.21.15.118:0/4234952457 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f15fc010c00 con 0x7f160807acd0 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.510+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f160810c8d0 con 0x7f15f8038590 2024-03-12T03:16:37.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.732+0000 7f1605ffb700 1 -- 172.21.15.118:0/4234952457 <== mgr.14164 v2:172.21.15.118:6800/180481365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f160810c8d0 con 0x7f15f8038590 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f15f8038590 msgr2=0x7f15f803aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f15f8038590 0x7f15f803aa50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f15f4006fd0 tx=0x7f15f4006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 msgr2=0x7f160819c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f160819c310 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f15fc0117b0 tx=0x7f15fc011890 comp rx=0 tx=0).stop 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 shutdown_connections 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f15f8038590 0x7f15f803aa50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 --2- 172.21.15.118:0/4234952457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f160807acd0 0x7f160819c310 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.734+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 >> 172.21.15.118:0/4234952457 conn(0x7f1608101900 msgr2=0x7f1608102500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.735+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 shutdown_connections 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:35.735+0000 7f160ee54700 1 -- 172.21.15.118:0/4234952457 wait complete. 2024-03-12T03:16:37.160 INFO:teuthology.orchestra.run.smithi118.stdout:Fetching dashboard port number... 2024-03-12T03:16:38.352 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 8443 2024-03-12T03:16:38.352 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e76e0700 1 Processor -- start 2024-03-12T03:16:38.352 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e76e0700 1 -- start start 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e76e0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e01091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e76e0700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e0079db0 con 0x7fc2e0106de0 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e547c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e01091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e547c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e01091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39968/0 (socket says 172.21.15.118:39968) 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e547c700 1 -- 172.21.15.118:0/2214758844 learned_addr learned my addr 172.21.15.118:0/2214758844 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.556+0000 7fc2e547c700 1 -- 172.21.15.118:0/2214758844 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e0109710 con 0x7fc2e0106de0 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2e547c700 1 --2- 172.21.15.118:0/2214758844 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e01091d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0098a0 tx=0x7fc2dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad95e531e1c796e server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2d7fff700 1 -- 172.21.15.118:0/2214758844 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2dc004180 con 0x7fc2e0106de0 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2d7fff700 1 -- 172.21.15.118:0/2214758844 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2dc0042e0 con 0x7fc2e0106de0 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2d7fff700 1 -- 172.21.15.118:0/2214758844 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2dc011b10 con 0x7fc2e0106de0 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2e76e0700 1 -- 172.21.15.118:0/2214758844 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 msgr2=0x7fc2e01091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2e76e0700 1 --2- 172.21.15.118:0/2214758844 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e01091d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0098a0 tx=0x7fc2dc009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2e76e0700 1 -- 172.21.15.118:0/2214758844 shutdown_connections 2024-03-12T03:16:38.353 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2e76e0700 1 --2- 172.21.15.118:0/2214758844 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e01091d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.557+0000 7fc2e76e0700 1 -- 172.21.15.118:0/2214758844 >> 172.21.15.118:0/2214758844 conn(0x7fc2e0100830 msgr2=0x7fc2e0102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e76e0700 1 -- 172.21.15.118:0/2214758844 shutdown_connections 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e76e0700 1 -- 172.21.15.118:0/2214758844 wait complete. 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e76e0700 1 Processor -- start 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e76e0700 1 -- start start 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e76e0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e0197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e76e0700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e01983e0 con 0x7fc2e0106de0 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e547c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e0197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e547c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e0197ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39982/0 (socket says 172.21.15.118:39982) 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.558+0000 7fc2e547c700 1 -- 172.21.15.118:0/1465625597 learned_addr learned my addr 172.21.15.118:0/1465625597 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.559+0000 7fc2e547c700 1 -- 172.21.15.118:0/1465625597 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc009550 con 0x7fc2e0106de0 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.559+0000 7fc2e547c700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e0197ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009fb0 tx=0x7fc2dc011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.559+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2dc0038b0 con 0x7fc2e0106de0 2024-03-12T03:16:38.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.559+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2dc003a10 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.559+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e01985e0 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.559+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2dc019750 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.559+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e0198a80 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.560+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 12) v1 ==== 45402+0+0 (secure 0 0 0) 0x7fc2dc0199c0 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.560+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e0191d60 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.560+0000 7fc2d67fc700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc2cc03c900 0x7fc2cc03edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.560+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc2dc04c7f0 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.560+0000 7fc2e4c7b700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc2cc03c900 0x7fc2cc03edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.561+0000 7fc2e4c7b700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc2cc03c900 0x7fc2cc03edc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0006fd0 tx=0x7fc2d0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.565+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc2dc010380 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.734+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7fc2dc01ac10 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.760+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fc2e002ccb0 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.760+0000 7fc2d67fc700 1 -- 172.21.15.118:0/1465625597 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fc2dc010380 con 0x7fc2e0106de0 2024-03-12T03:16:38.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc2cc03c900 msgr2=0x7fc2cc03edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc2cc03c900 0x7fc2cc03edc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc2d0006fd0 tx=0x7fc2d0006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 msgr2=0x7fc2e0197ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e0197ea0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009fb0 tx=0x7fc2dc011890 comp rx=0 tx=0).stop 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 shutdown_connections 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc2cc03c900 0x7fc2cc03edc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 --2- 172.21.15.118:0/1465625597 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2e0106de0 0x7fc2e0197ea0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 >> 172.21.15.118:0/1465625597 conn(0x7fc2e0100830 msgr2=0x7fc2e01011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 shutdown_connections 2024-03-12T03:16:38.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:37.762+0000 7fc2e76e0700 1 -- 172.21.15.118:0/1465625597 wait complete. 2024-03-12T03:16:38.359 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-12T03:16:38.359 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-12T03:16:38.366 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-12T03:16:38.366 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-12T03:16:38.366 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-12T03:16:38.367 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T03:16:38.368 INFO:teuthology.orchestra.run.smithi118.stdout:Ceph Dashboard is now available at: 2024-03-12T03:16:38.368 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:38.368 INFO:teuthology.orchestra.run.smithi118.stdout: URL: https://smithi118.front.sepia.ceph.com:8443/ 2024-03-12T03:16:38.368 INFO:teuthology.orchestra.run.smithi118.stdout: User: admin 2024-03-12T03:16:38.369 INFO:teuthology.orchestra.run.smithi118.stdout: Password: u907t0kfgp 2024-03-12T03:16:38.369 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:38.369 INFO:teuthology.orchestra.run.smithi118.stdout:Saving cluster configuration to /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config directory 2024-03-12T03:16:38.369 INFO:teuthology.orchestra.run.smithi118.stdout:Enabling autotune for osd_memory_target 2024-03-12T03:16:38.991 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:38 smithi118 ceph-mon[29160]: mgrmap e13: smithi118.zzzuvl(active, since 6s) 2024-03-12T03:16:38.991 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:38 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1465625597' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T03:16:39.558 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.787+0000 7f1d29d66700 1 Processor -- start 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.787+0000 7f1d29d66700 1 -- start start 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.787+0000 7f1d29d66700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d24109200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.787+0000 7f1d29d66700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d24079d90 con 0x7f1d24106e10 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d237fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d24109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d237fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d24109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39992/0 (socket says 172.21.15.118:39992) 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d237fe700 1 -- 172.21.15.118:0/926049350 learned_addr learned my addr 172.21.15.118:0/926049350 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d237fe700 1 -- 172.21.15.118:0/926049350 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d24109740 con 0x7f1d24106e10 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d237fe700 1 --2- 172.21.15.118:0/926049350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d24109200 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1d180098a0 tx=0x7f1d18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cc79f69b872a5f server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d227fc700 1 -- 172.21.15.118:0/926049350 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d18004180 con 0x7f1d24106e10 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d227fc700 1 -- 172.21.15.118:0/926049350 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d180042e0 con 0x7f1d24106e10 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.788+0000 7f1d227fc700 1 -- 172.21.15.118:0/926049350 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d18011b10 con 0x7f1d24106e10 2024-03-12T03:16:39.559 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 -- 172.21.15.118:0/926049350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 msgr2=0x7f1d24109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 --2- 172.21.15.118:0/926049350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d24109200 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1d180098a0 tx=0x7f1d18009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 -- 172.21.15.118:0/926049350 shutdown_connections 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 --2- 172.21.15.118:0/926049350 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d24109200 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 -- 172.21.15.118:0/926049350 >> 172.21.15.118:0/926049350 conn(0x7f1d24100840 msgr2=0x7f1d24102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 -- 172.21.15.118:0/926049350 shutdown_connections 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 -- 172.21.15.118:0/926049350 wait complete. 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.789+0000 7f1d29d66700 1 Processor -- start 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d29d66700 1 -- start start 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d29d66700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d2419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d29d66700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2419a650 con 0x7f1d24106e10 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d237fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d2419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d237fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d2419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40000/0 (socket says 172.21.15.118:40000) 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d237fe700 1 -- 172.21.15.118:0/3109757473 learned_addr learned my addr 172.21.15.118:0/3109757473 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d237fe700 1 -- 172.21.15.118:0/3109757473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d18009550 con 0x7f1d24106e10 2024-03-12T03:16:39.560 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d237fe700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d2419a110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d18011670 tx=0x7f1d18011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.790+0000 7f1d20ff9700 1 -- 172.21.15.118:0/3109757473 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d180038b0 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.791+0000 7f1d20ff9700 1 -- 172.21.15.118:0/3109757473 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d18003a10 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.791+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2419a850 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.791+0000 7f1d20ff9700 1 -- 172.21.15.118:0/3109757473 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d1801a7c0 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.791+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2419acf0 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.791+0000 7f1d20ff9700 1 -- 172.21.15.118:0/3109757473 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f1d1801a980 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.791+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2410a4d0 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.792+0000 7f1d20ff9700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1d0c0385f0 0x7f1d0c03aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.792+0000 7f1d20ff9700 1 -- 172.21.15.118:0/3109757473 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1d1804c8d0 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.792+0000 7f1d22ffd700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1d0c0385f0 0x7f1d0c03aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.792+0000 7f1d22ffd700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1d0c0385f0 0x7f1d0c03aab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d14006fd0 tx=0x7f1d14006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.796+0000 7f1d20ff9700 1 -- 172.21.15.118:0/3109757473 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1d18022330 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.988+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f1d2410a6c0 con 0x7f1d24106e10 2024-03-12T03:16:39.561 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.989+0000 7f1d20ff9700 1 -- 172.21.15.118:0/3109757473 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1d18019e40 con 0x7f1d24106e10 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1d0c0385f0 msgr2=0x7f1d0c03aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1d0c0385f0 0x7f1d0c03aab0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d14006fd0 tx=0x7f1d14006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 msgr2=0x7f1d2419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d2419a110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d18011670 tx=0x7f1d18011890 comp rx=0 tx=0).stop 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 shutdown_connections 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1d0c0385f0 0x7f1d0c03aab0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 --2- 172.21.15.118:0/3109757473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d24106e10 0x7f1d2419a110 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 >> 172.21.15.118:0/3109757473 conn(0x7f1d24100840 msgr2=0x7f1d24101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 shutdown_connections 2024-03-12T03:16:39.562 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:38.991+0000 7f1d29d66700 1 -- 172.21.15.118:0/3109757473 wait complete. 2024-03-12T03:16:40.848 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5122a95700 1 Processor -- start 2024-03-12T03:16:40.848 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5122a95700 1 -- start start 2024-03-12T03:16:40.848 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5122a95700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c1091f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:40.848 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5122a95700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c079d90 con 0x7f511c106e00 2024-03-12T03:16:40.848 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5120831700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c1091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5120831700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c1091f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40008/0 (socket says 172.21.15.118:40008) 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5120831700 1 -- 172.21.15.118:0/339023478 learned_addr learned my addr 172.21.15.118:0/339023478 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.992+0000 7f5120831700 1 -- 172.21.15.118:0/339023478 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c109730 con 0x7f511c106e00 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5120831700 1 --2- 172.21.15.118:0/339023478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c1091f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51100098a0 tx=0x7f5110009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ae3214dffdaa75 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f511b7fe700 1 -- 172.21.15.118:0/339023478 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5110004180 con 0x7f511c106e00 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f511b7fe700 1 -- 172.21.15.118:0/339023478 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51100042e0 con 0x7f511c106e00 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f511b7fe700 1 -- 172.21.15.118:0/339023478 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5110011b10 con 0x7f511c106e00 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5122a95700 1 -- 172.21.15.118:0/339023478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 msgr2=0x7f511c1091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5122a95700 1 --2- 172.21.15.118:0/339023478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c1091f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51100098a0 tx=0x7f5110009bb0 comp rx=0 tx=0).stop 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5122a95700 1 -- 172.21.15.118:0/339023478 shutdown_connections 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5122a95700 1 --2- 172.21.15.118:0/339023478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c1091f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:40.849 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5122a95700 1 -- 172.21.15.118:0/339023478 >> 172.21.15.118:0/339023478 conn(0x7f511c1007f0 msgr2=0x7f511c102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5122a95700 1 -- 172.21.15.118:0/339023478 shutdown_connections 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.993+0000 7f5122a95700 1 -- 172.21.15.118:0/339023478 wait complete. 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5122a95700 1 Processor -- start 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5122a95700 1 -- start start 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5122a95700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5122a95700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c19a5c0 con 0x7f511c106e00 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5120831700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5120831700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c19a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40018/0 (socket says 172.21.15.118:40018) 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5120831700 1 -- 172.21.15.118:0/2226153508 learned_addr learned my addr 172.21.15.118:0/2226153508 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.994+0000 7f5120831700 1 -- 172.21.15.118:0/2226153508 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110009550 con 0x7f511c106e00 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.995+0000 7f5120831700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c19a080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5110009fb0 tx=0x7f5110011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.995+0000 7f5119ffb700 1 -- 172.21.15.118:0/2226153508 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51100038b0 con 0x7f511c106e00 2024-03-12T03:16:40.850 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.995+0000 7f5119ffb700 1 -- 172.21.15.118:0/2226153508 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5110003a10 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.995+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c19a7c0 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.995+0000 7f5119ffb700 1 -- 172.21.15.118:0/2226153508 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5110019750 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.995+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c19ac60 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.996+0000 7f5119ffb700 1 -- 172.21.15.118:0/2226153508 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f51100198b0 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.996+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511c10a4c0 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.996+0000 7f5119ffb700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f510c0385f0 0x7f510c03aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.996+0000 7f5119ffb700 1 -- 172.21.15.118:0/2226153508 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f511004c6e0 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.996+0000 7f511bfff700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f510c0385f0 0x7f510c03aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:39.996+0000 7f511bfff700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f510c0385f0 0x7f510c03aab0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5108006fd0 tx=0x7f5108006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.001+0000 7f5119ffb700 1 -- 172.21.15.118:0/2226153508 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5110022d70 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.300+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f511c10a6b0 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.303+0000 7f5119ffb700 1 -- 172.21.15.118:0/2226153508 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7f5110022310 con 0x7f511c106e00 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f510c0385f0 msgr2=0x7f510c03aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:40.851 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f510c0385f0 0x7f510c03aab0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5108006fd0 tx=0x7f5108006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 msgr2=0x7f511c19a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c19a080 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5110009fb0 tx=0x7f5110011890 comp rx=0 tx=0).stop 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 shutdown_connections 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f510c0385f0 0x7f510c03aab0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 --2- 172.21.15.118:0/2226153508 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c106e00 0x7f511c19a080 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 >> 172.21.15.118:0/2226153508 conn(0x7f511c1007f0 msgr2=0x7f511c101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 shutdown_connections 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-12T03:16:40.305+0000 7f5122a95700 1 -- 172.21.15.118:0/2226153508 wait complete. 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:16:40.852 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout:Or, if you are only running a single cluster on this host: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: ceph telemetry on 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout:For more information see: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:40.853 INFO:teuthology.orchestra.run.smithi118.stdout:Bootstrap complete. 2024-03-12T03:16:40.881 INFO:tasks.cephadm:Fetching config... 2024-03-12T03:16:40.881 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:16:40.881 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T03:16:40.896 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T03:16:40.896 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:16:40.896 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T03:16:40.950 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T03:16:40.950 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:16:40.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/keyring of=/dev/stdout 2024-03-12T03:16:41.018 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T03:16:41.018 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:16:41.018 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T03:16:41.073 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T03:16:41.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ZpAcWYiwBVeEkSS1JXdHkfbiOhNSplrlpqUzMe2B2dBA6kvvGYVParX72/s/yVTXBrASTWD5CVxc60ioJfitnieEHayZvLfzYsejyJzBsHQ34Sigs36PgTyzG7Cbt23+gF8fJyN4p8DomI09iTgeLFk3nxJQP0/kNCnhXWY92gIy1qfyyJhl7c8iMOlG646zWrXABNBCD55oPtiaCS2B1d8I0ZAU75iCZesrkXVOeqKXCLIHzeXwBzDdzp0Cmf6Nprwup4XDr4I77nFzU+FkRYf5IVN/ybQC/oPm/eE8GN0T//uQEVrELRGVg/KItwBzg+IzZEfHkpsAgkFJRu+G6JAjly9cGl0wF5/caHuir0LlHgegZK/B+kIB+yugFbg7ose1VqNMs3CogY/2jnn/r16oJcvB+NnEOs0gmL2YnGNKMcD8QFSGEjpF3B2OztAC8q2zjLYMUEkIsQ3/a6JPq1GLIap9cDoeJJ+cpeqQmj/mTaIzxdeuxa0z4qSPneE= ceph-9d8e7844-e01e-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:16:41.151 INFO:teuthology.orchestra.run.smithi118.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ZpAcWYiwBVeEkSS1JXdHkfbiOhNSplrlpqUzMe2B2dBA6kvvGYVParX72/s/yVTXBrASTWD5CVxc60ioJfitnieEHayZvLfzYsejyJzBsHQ34Sigs36PgTyzG7Cbt23+gF8fJyN4p8DomI09iTgeLFk3nxJQP0/kNCnhXWY92gIy1qfyyJhl7c8iMOlG646zWrXABNBCD55oPtiaCS2B1d8I0ZAU75iCZesrkXVOeqKXCLIHzeXwBzDdzp0Cmf6Nprwup4XDr4I77nFzU+FkRYf5IVN/ybQC/oPm/eE8GN0T//uQEVrELRGVg/KItwBzg+IzZEfHkpsAgkFJRu+G6JAjly9cGl0wF5/caHuir0LlHgegZK/B+kIB+yugFbg7ose1VqNMs3CogY/2jnn/r16oJcvB+NnEOs0gmL2YnGNKMcD8QFSGEjpF3B2OztAC8q2zjLYMUEkIsQ3/a6JPq1GLIap9cDoeJJ+cpeqQmj/mTaIzxdeuxa0z4qSPneE= ceph-9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:16:41.165 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ZpAcWYiwBVeEkSS1JXdHkfbiOhNSplrlpqUzMe2B2dBA6kvvGYVParX72/s/yVTXBrASTWD5CVxc60ioJfitnieEHayZvLfzYsejyJzBsHQ34Sigs36PgTyzG7Cbt23+gF8fJyN4p8DomI09iTgeLFk3nxJQP0/kNCnhXWY92gIy1qfyyJhl7c8iMOlG646zWrXABNBCD55oPtiaCS2B1d8I0ZAU75iCZesrkXVOeqKXCLIHzeXwBzDdzp0Cmf6Nprwup4XDr4I77nFzU+FkRYf5IVN/ybQC/oPm/eE8GN0T//uQEVrELRGVg/KItwBzg+IzZEfHkpsAgkFJRu+G6JAjly9cGl0wF5/caHuir0LlHgegZK/B+kIB+yugFbg7ose1VqNMs3CogY/2jnn/r16oJcvB+NnEOs0gmL2YnGNKMcD8QFSGEjpF3B2OztAC8q2zjLYMUEkIsQ3/a6JPq1GLIap9cDoeJJ+cpeqQmj/mTaIzxdeuxa0z4qSPneE= ceph-9d8e7844-e01e-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:16:41.211 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0ZpAcWYiwBVeEkSS1JXdHkfbiOhNSplrlpqUzMe2B2dBA6kvvGYVParX72/s/yVTXBrASTWD5CVxc60ioJfitnieEHayZvLfzYsejyJzBsHQ34Sigs36PgTyzG7Cbt23+gF8fJyN4p8DomI09iTgeLFk3nxJQP0/kNCnhXWY92gIy1qfyyJhl7c8iMOlG646zWrXABNBCD55oPtiaCS2B1d8I0ZAU75iCZesrkXVOeqKXCLIHzeXwBzDdzp0Cmf6Nprwup4XDr4I77nFzU+FkRYf5IVN/ybQC/oPm/eE8GN0T//uQEVrELRGVg/KItwBzg+IzZEfHkpsAgkFJRu+G6JAjly9cGl0wF5/caHuir0LlHgegZK/B+kIB+yugFbg7ose1VqNMs3CogY/2jnn/r16oJcvB+NnEOs0gmL2YnGNKMcD8QFSGEjpF3B2OztAC8q2zjLYMUEkIsQ3/a6JPq1GLIap9cDoeJJ+cpeqQmj/mTaIzxdeuxa0z4qSPneE= ceph-9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:16:41.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T03:16:41.482 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:16:41.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:41 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2226153508' entity='client.admin' 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.027+0000 7f787fd32700 1 -- 172.21.15.118:0/152010765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 msgr2=0x7f7878102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.027+0000 7f787fd32700 1 --2- 172.21.15.118:0/152010765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878102130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7874009910 tx=0x7f7874009c20 comp rx=0 tx=0).stop 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.027+0000 7f787fd32700 1 -- 172.21.15.118:0/152010765 shutdown_connections 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.027+0000 7f787fd32700 1 --2- 172.21.15.118:0/152010765 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878102130 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.027+0000 7f787fd32700 1 -- 172.21.15.118:0/152010765 >> 172.21.15.118:0/152010765 conn(0x7f78780fa370 msgr2=0x7f78780fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787fd32700 1 -- 172.21.15.118:0/152010765 shutdown_connections 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787fd32700 1 -- 172.21.15.118:0/152010765 wait complete. 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787fd32700 1 Processor -- start 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787fd32700 1 -- start start 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787fd32700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787fd32700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78781960b0 con 0x7f7878101d10 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787dace700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787dace700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40036/0 (socket says 172.21.15.118:40036) 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.028+0000 7f787dace700 1 -- 172.21.15.118:0/1862338935 learned_addr learned my addr 172.21.15.118:0/1862338935 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.029+0000 7f787dace700 1 -- 172.21.15.118:0/1862338935 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78740095f0 con 0x7f7878101d10 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.029+0000 7f787dace700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878195b70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7874006010 tx=0x7f7874009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.029+0000 7f786effd700 1 -- 172.21.15.118:0/1862338935 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f787401d070 con 0x7f7878101d10 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.029+0000 7f786effd700 1 -- 172.21.15.118:0/1862338935 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7874005dc0 con 0x7f7878101d10 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.029+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78781962b0 con 0x7f7878101d10 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.029+0000 7f786effd700 1 -- 172.21.15.118:0/1862338935 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7874022e90 con 0x7f7878101d10 2024-03-12T03:16:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.029+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7878196750 con 0x7f7878101d10 2024-03-12T03:16:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.030+0000 7f786effd700 1 -- 172.21.15.118:0/1862338935 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f787400f4f0 con 0x7f7878101d10 2024-03-12T03:16:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.030+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787818f560 con 0x7f7878101d10 2024-03-12T03:16:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.030+0000 7f786effd700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7864038530 0x7f786403a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.030+0000 7f786effd700 1 -- 172.21.15.118:0/1862338935 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f787404c4f0 con 0x7f7878101d10 2024-03-12T03:16:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.030+0000 7f787d2cd700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7864038530 0x7f786403a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:44.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.031+0000 7f787d2cd700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7864038530 0x7f786403a9f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7868006fd0 tx=0x7f7868006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:44.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.035+0000 7f786effd700 1 -- 172.21.15.118:0/1862338935 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7874022750 con 0x7f7878101d10 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi118", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi118", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:16:44.177 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:43 smithi118 ceph-mon[29160]: Deploying daemon ceph-exporter.smithi118 on smithi118 2024-03-12T03:16:44.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.232+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f78780623e0 con 0x7f7878101d10 2024-03-12T03:16:44.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.235+0000 7f786effd700 1 -- 172.21.15.118:0/1862338935 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f78780623e0 con 0x7f7878101d10 2024-03-12T03:16:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7864038530 msgr2=0x7f786403a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7864038530 0x7f786403a9f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7868006fd0 tx=0x7f7868006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 msgr2=0x7f7878195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878195b70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7874006010 tx=0x7f7874009f80 comp rx=0 tx=0).stop 2024-03-12T03:16:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 shutdown_connections 2024-03-12T03:16:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7864038530 0x7f786403a9f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 --2- 172.21.15.118:0/1862338935 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7878101d10 0x7f7878195b70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.237+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 >> 172.21.15.118:0/1862338935 conn(0x7f78780fa370 msgr2=0x7f78780fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.238+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 shutdown_connections 2024-03-12T03:16:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:44.238+0000 7f787fd32700 1 -- 172.21.15.118:0/1862338935 wait complete. 2024-03-12T03:16:44.985 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T03:16:44.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T03:16:45.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:45 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1862338935' entity='client.admin' 2024-03-12T03:16:45.652 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:16:47.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.692+0000 7fc3653f4700 1 -- 172.21.15.118:0/700009164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 msgr2=0x7fc360103560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:47.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.692+0000 7fc3653f4700 1 --2- 172.21.15.118:0/700009164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360103560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc350009910 tx=0x7fc350009c20 comp rx=0 tx=0).stop 2024-03-12T03:16:47.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.692+0000 7fc3653f4700 1 -- 172.21.15.118:0/700009164 shutdown_connections 2024-03-12T03:16:47.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.692+0000 7fc3653f4700 1 --2- 172.21.15.118:0/700009164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360103560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:47.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.692+0000 7fc3653f4700 1 -- 172.21.15.118:0/700009164 >> 172.21.15.118:0/700009164 conn(0x7fc3600fab80 msgr2=0x7fc3600fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:47.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.692+0000 7fc3653f4700 1 -- 172.21.15.118:0/700009164 shutdown_connections 2024-03-12T03:16:47.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.692+0000 7fc3653f4700 1 -- 172.21.15.118:0/700009164 wait complete. 2024-03-12T03:16:47.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc3653f4700 1 Processor -- start 2024-03-12T03:16:47.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc3653f4700 1 -- start start 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc3653f4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc3653f4700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc360196090 con 0x7fc360101170 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc35effd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc35effd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360195b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51060/0 (socket says 172.21.15.118:51060) 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc35effd700 1 -- 172.21.15.118:0/794706221 learned_addr learned my addr 172.21.15.118:0/794706221 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.693+0000 7fc35effd700 1 -- 172.21.15.118:0/794706221 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3500095f0 con 0x7fc360101170 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.694+0000 7fc35effd700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360195b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc350009ed0 tx=0x7fc350004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.694+0000 7fc35cff9700 1 -- 172.21.15.118:0/794706221 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc35001d070 con 0x7fc360101170 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.694+0000 7fc35cff9700 1 -- 172.21.15.118:0/794706221 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc35000be20 con 0x7fc360101170 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.694+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc360196290 con 0x7fc360101170 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.694+0000 7fc35cff9700 1 -- 172.21.15.118:0/794706221 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc35000f460 con 0x7fc360101170 2024-03-12T03:16:47.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.694+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc360196730 con 0x7fc360101170 2024-03-12T03:16:47.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.695+0000 7fc35cff9700 1 -- 172.21.15.118:0/794706221 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7fc35000f5c0 con 0x7fc360101170 2024-03-12T03:16:47.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.695+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36018f510 con 0x7fc360101170 2024-03-12T03:16:47.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.695+0000 7fc35cff9700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc34c0385f0 0x7fc34c03aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:47.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.695+0000 7fc35cff9700 1 -- 172.21.15.118:0/794706221 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc35004c770 con 0x7fc360101170 2024-03-12T03:16:47.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.695+0000 7fc3567fc700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc34c0385f0 0x7fc34c03aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:47.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.695+0000 7fc3567fc700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc34c0385f0 0x7fc34c03aab0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc348006fd0 tx=0x7fc348006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:47.702 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.699+0000 7fc35cff9700 1 -- 172.21.15.118:0/794706221 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc35002a360 con 0x7fc360101170 2024-03-12T03:16:47.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.899+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3600611f0 con 0x7fc34c0385f0 2024-03-12T03:16:47.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.902+0000 7fc35cff9700 1 -- 172.21.15.118:0/794706221 <== mgr.14164 v2:172.21.15.118:6800/180481365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc3600611f0 con 0x7fc34c0385f0 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc34c0385f0 msgr2=0x7fc34c03aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc34c0385f0 0x7fc34c03aab0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc348006fd0 tx=0x7fc348006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 msgr2=0x7fc360195b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360195b50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc350009ed0 tx=0x7fc350004690 comp rx=0 tx=0).stop 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 shutdown_connections 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fc34c0385f0 0x7fc34c03aab0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 --2- 172.21.15.118:0/794706221 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc360101170 0x7fc360195b50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:47.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 >> 172.21.15.118:0/794706221 conn(0x7fc3600fab80 msgr2=0x7fc3600fb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:47.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 shutdown_connections 2024-03-12T03:16:47.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:47.904+0000 7fc3653f4700 1 -- 172.21.15.118:0/794706221 wait complete. 2024-03-12T03:16:48.758 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-03-12T03:16:48.758 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:16:48.758 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-03-12T03:16:48.776 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:16:48.776 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:16:48.834 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-03-12T03:16:48.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch host add smithi177 2024-03-12T03:16:49.123 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:48 smithi118 ceph-mon[29160]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:49.124 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:48 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:49.356 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:16:50.623 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:50.623 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:50.623 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:50.623 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:50.623 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi118", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-12T03:16:50.623 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi118", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-12T03:16:50.623 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:16:50.624 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:50 smithi118 ceph-mon[29160]: Deploying daemon crash.smithi118 on smithi118 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.850+0000 7efe7dd71700 1 -- 172.21.15.118:0/938496220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 msgr2=0x7efe780ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.850+0000 7efe7dd71700 1 --2- 172.21.15.118:0/938496220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe780ffa00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efe68009910 tx=0x7efe68009c20 comp rx=0 tx=0).stop 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.850+0000 7efe7dd71700 1 -- 172.21.15.118:0/938496220 shutdown_connections 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.850+0000 7efe7dd71700 1 --2- 172.21.15.118:0/938496220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe780ffa00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.850+0000 7efe7dd71700 1 -- 172.21.15.118:0/938496220 >> 172.21.15.118:0/938496220 conn(0x7efe780fab80 msgr2=0x7efe780fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.850+0000 7efe7dd71700 1 -- 172.21.15.118:0/938496220 shutdown_connections 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.850+0000 7efe7dd71700 1 -- 172.21.15.118:0/938496220 wait complete. 2024-03-12T03:16:50.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.851+0000 7efe7dd71700 1 Processor -- start 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.851+0000 7efe7dd71700 1 -- start start 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.851+0000 7efe7dd71700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe78069130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.851+0000 7efe7dd71700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe78069670 con 0x7efe780ff5e0 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.851+0000 7efe777fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe78069130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.851+0000 7efe777fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe78069130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51098/0 (socket says 172.21.15.118:51098) 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.851+0000 7efe777fe700 1 -- 172.21.15.118:0/4013757753 learned_addr learned my addr 172.21.15.118:0/4013757753 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.852+0000 7efe777fe700 1 -- 172.21.15.118:0/4013757753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680095f0 con 0x7efe780ff5e0 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.852+0000 7efe777fe700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe78069130 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efe68006010 tx=0x7efe68009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.852+0000 7efe74ff9700 1 -- 172.21.15.118:0/4013757753 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe6801d070 con 0x7efe780ff5e0 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.852+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe781988d0 con 0x7efe780ff5e0 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.852+0000 7efe74ff9700 1 -- 172.21.15.118:0/4013757753 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe68005dc0 con 0x7efe780ff5e0 2024-03-12T03:16:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.852+0000 7efe74ff9700 1 -- 172.21.15.118:0/4013757753 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe68022e90 con 0x7efe780ff5e0 2024-03-12T03:16:50.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.852+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe78198d10 con 0x7efe780ff5e0 2024-03-12T03:16:50.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.853+0000 7efe74ff9700 1 -- 172.21.15.118:0/4013757753 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7efe6800f4d0 con 0x7efe780ff5e0 2024-03-12T03:16:50.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.853+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7818e410 con 0x7efe780ff5e0 2024-03-12T03:16:50.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.853+0000 7efe74ff9700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7efe60038590 0x7efe6003aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:50.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.853+0000 7efe74ff9700 1 -- 172.21.15.118:0/4013757753 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efe68022470 con 0x7efe780ff5e0 2024-03-12T03:16:50.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.853+0000 7efe76ffd700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7efe60038590 0x7efe6003aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:50.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.854+0000 7efe76ffd700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7efe60038590 0x7efe6003aa50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe6c006fd0 tx=0x7efe6c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:50.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:50.862+0000 7efe74ff9700 1 -- 172.21.15.118:0/4013757753 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7efe68017440 con 0x7efe780ff5e0 2024-03-12T03:16:51.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:51.071+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}) v1 -- 0x7efe780611f0 con 0x7efe60038590 2024-03-12T03:16:52.800 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:52 smithi118 ceph-mon[29160]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:16:52.800 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:52 smithi118 ceph-mon[29160]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:52.801 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:52 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:52.801 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:52 smithi118 ceph-mon[29160]: Deploying cephadm binary to smithi177 2024-03-12T03:16:53.799 INFO:teuthology.orchestra.run.smithi118.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-03-12T03:16:53.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.796+0000 7efe74ff9700 1 -- 172.21.15.118:0/4013757753 <== mgr.14164 v2:172.21.15.118:6800/180481365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7efe780611f0 con 0x7efe60038590 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7efe60038590 msgr2=0x7efe6003aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7efe60038590 0x7efe6003aa50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe6c006fd0 tx=0x7efe6c006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 msgr2=0x7efe78069130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe78069130 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7efe68006010 tx=0x7efe68009f80 comp rx=0 tx=0).stop 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 shutdown_connections 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7efe60038590 0x7efe6003aa50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 --2- 172.21.15.118:0/4013757753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efe780ff5e0 0x7efe78069130 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 >> 172.21.15.118:0/4013757753 conn(0x7efe780fab80 msgr2=0x7efe780fb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 shutdown_connections 2024-03-12T03:16:53.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:53.798+0000 7efe7dd71700 1 -- 172.21.15.118:0/4013757753 wait complete. 2024-03-12T03:16:54.308 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:54.308 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:54.308 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:54.308 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:54.308 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: Deploying daemon node-exporter.smithi118 on smithi118 2024-03-12T03:16:54.308 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:54.308 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:54.309 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:54 smithi118 ceph-mon[29160]: Added host smithi177 2024-03-12T03:16:54.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T03:16:54.682 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:16:56.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:56 smithi118 ceph-mon[29160]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:56.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:56 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:57.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.095+0000 7f4186a9c700 1 -- 172.21.15.118:0/255846732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 msgr2=0x7f4180100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:57.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.095+0000 7f4186a9c700 1 --2- 172.21.15.118:0/255846732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180100b10 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4174009910 tx=0x7f4174009c20 comp rx=0 tx=0).stop 2024-03-12T03:16:57.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.095+0000 7f4186a9c700 1 -- 172.21.15.118:0/255846732 shutdown_connections 2024-03-12T03:16:57.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.095+0000 7f4186a9c700 1 --2- 172.21.15.118:0/255846732 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180100b10 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:57.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.095+0000 7f4186a9c700 1 -- 172.21.15.118:0/255846732 >> 172.21.15.118:0/255846732 conn(0x7f41800fbcb0 msgr2=0x7f41800fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.095+0000 7f4186a9c700 1 -- 172.21.15.118:0/255846732 shutdown_connections 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.095+0000 7f4186a9c700 1 -- 172.21.15.118:0/255846732 wait complete. 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4186a9c700 1 Processor -- start 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4186a9c700 1 -- start start 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4186a9c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4186a9c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41801960b0 con 0x7f41801006f0 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4184838700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4184838700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45494/0 (socket says 172.21.15.118:45494) 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4184838700 1 -- 172.21.15.118:0/1834371250 learned_addr learned my addr 172.21.15.118:0/1834371250 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.096+0000 7f4184838700 1 -- 172.21.15.118:0/1834371250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41740095f0 con 0x7f41801006f0 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.097+0000 7f4184838700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180195b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4174006010 tx=0x7f4174009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.097+0000 7f417dffb700 1 -- 172.21.15.118:0/1834371250 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f417401d070 con 0x7f41801006f0 2024-03-12T03:16:57.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.097+0000 7f417dffb700 1 -- 172.21.15.118:0/1834371250 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4174005dc0 con 0x7f41801006f0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.097+0000 7f417dffb700 1 -- 172.21.15.118:0/1834371250 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4174022e90 con 0x7f41801006f0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.097+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41801962b0 con 0x7f41801006f0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.097+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4180196750 con 0x7f41801006f0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.098+0000 7f417dffb700 1 -- 172.21.15.118:0/1834371250 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f4174022470 con 0x7f41801006f0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.098+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418018f520 con 0x7f41801006f0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.098+0000 7f417dffb700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4170038530 0x7f417003a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.098+0000 7f417dffb700 1 -- 172.21.15.118:0/1834371250 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f417404c640 con 0x7f41801006f0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.098+0000 7f417ffff700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4170038530 0x7f417003a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.098+0000 7f417ffff700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4170038530 0x7f417003a9f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f416c006fd0 tx=0x7f416c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:16:57.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.103+0000 7f417dffb700 1 -- 172.21.15.118:0/1834371250 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f417400b960 con 0x7f41801006f0 2024-03-12T03:16:57.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.302+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41800611f0 con 0x7f4170038530 2024-03-12T03:16:57.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.304+0000 7f417dffb700 1 -- 172.21.15.118:0/1834371250 <== mgr.14164 v2:172.21.15.118:6800/180481365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f41800611f0 con 0x7f4170038530 2024-03-12T03:16:57.306 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:16:57.306 INFO:teuthology.orchestra.run.smithi118.stdout:[{"addr": "172.21.15.118", "hostname": "smithi118", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-03-12T03:16:57.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4170038530 msgr2=0x7f417003a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:57.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4170038530 0x7f417003a9f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f416c006fd0 tx=0x7f416c006e40 comp rx=0 tx=0).stop 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 msgr2=0x7f4180195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180195b70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4174006010 tx=0x7f4174009f80 comp rx=0 tx=0).stop 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 shutdown_connections 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4170038530 0x7f417003a9f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 --2- 172.21.15.118:0/1834371250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41801006f0 0x7f4180195b70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 >> 172.21.15.118:0/1834371250 conn(0x7f41800fbcb0 msgr2=0x7f41800fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 shutdown_connections 2024-03-12T03:16:57.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:16:57.306+0000 7f4186a9c700 1 -- 172.21.15.118:0/1834371250 wait complete. 2024-03-12T03:16:58.183 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T03:16:58.184 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T03:16:58.680 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:16:58.803 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:58 smithi118 ceph-mon[29160]: from='client.14193 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:16:58.803 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:58 smithi118 ceph-mon[29160]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:16:58.803 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:58 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:58.804 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:58 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:58.804 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:58 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:58.804 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:58 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:16:59.925 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:16:59 smithi118 ceph-mon[29160]: Deploying daemon alertmanager.smithi118 on smithi118 2024-03-12T03:17:00.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.148+0000 7f047a65e700 1 -- 172.21.15.118:0/3561509679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 msgr2=0x7f0474100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.148+0000 7f047a65e700 1 --2- 172.21.15.118:0/3561509679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474100b10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0468009910 tx=0x7f0468009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.148+0000 7f047a65e700 1 -- 172.21.15.118:0/3561509679 shutdown_connections 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.148+0000 7f047a65e700 1 --2- 172.21.15.118:0/3561509679 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474100b10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.148+0000 7f047a65e700 1 -- 172.21.15.118:0/3561509679 >> 172.21.15.118:0/3561509679 conn(0x7f04740fbc90 msgr2=0x7f04740fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.148+0000 7f047a65e700 1 -- 172.21.15.118:0/3561509679 shutdown_connections 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.148+0000 7f047a65e700 1 -- 172.21.15.118:0/3561509679 wait complete. 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f047a65e700 1 Processor -- start 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f047a65e700 1 -- start start 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f047a65e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f047a65e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04741960d0 con 0x7f04741006f0 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f0473fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f0473fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45512/0 (socket says 172.21.15.118:45512) 2024-03-12T03:17:00.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f0473fff700 1 -- 172.21.15.118:0/980716675 learned_addr learned my addr 172.21.15.118:0/980716675 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.149+0000 7f0473fff700 1 -- 172.21.15.118:0/980716675 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680095f0 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.150+0000 7f0473fff700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474195b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f04680050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.150+0000 7f04717fa700 1 -- 172.21.15.118:0/980716675 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f046801d070 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.150+0000 7f04717fa700 1 -- 172.21.15.118:0/980716675 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f046800be20 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.150+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04741962d0 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.150+0000 7f04717fa700 1 -- 172.21.15.118:0/980716675 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0468022de0 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.150+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0474196770 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.151+0000 7f04717fa700 1 -- 172.21.15.118:0/980716675 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f046800f460 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.151+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047418f520 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.151+0000 7f04717fa700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f045c0385f0 0x7f045c03aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.151+0000 7f04717fa700 1 -- 172.21.15.118:0/980716675 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f046804c570 con 0x7f04741006f0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.151+0000 7f04737fe700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f045c0385f0 0x7f045c03aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.151+0000 7f04737fe700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f045c0385f0 0x7f045c03aab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0464006fd0 tx=0x7f0464006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:00.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.156+0000 7f04717fa700 1 -- 172.21.15.118:0/980716675 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0468017440 con 0x7f04741006f0 2024-03-12T03:17:00.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.382+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f04740623e0 con 0x7f04741006f0 2024-03-12T03:17:00.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.814+0000 7f04717fa700 1 -- 172.21.15.118:0/980716675 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0468027070 con 0x7f04741006f0 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f045c0385f0 msgr2=0x7f045c03aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f045c0385f0 0x7f045c03aab0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0464006fd0 tx=0x7f0464006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 msgr2=0x7f0474195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474195b90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f04680050b0 comp rx=0 tx=0).stop 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 shutdown_connections 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f045c0385f0 0x7f045c03aab0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 --2- 172.21.15.118:0/980716675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f04741006f0 0x7f0474195b90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 >> 172.21.15.118:0/980716675 conn(0x7f04740fbc90 msgr2=0x7f04740fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 shutdown_connections 2024-03-12T03:17:00.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:17:00.816+0000 7f047a65e700 1 -- 172.21.15.118:0/980716675 wait complete. 2024-03-12T03:17:00.820 INFO:teuthology.orchestra.run.smithi118.stderr:adjusted tunables profile to default 2024-03-12T03:17:01.238 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:00 smithi118 ceph-mon[29160]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:01.239 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:00 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/980716675' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:17:01.719 INFO:tasks.cephadm:Adding mon.smithi118 on smithi118 2024-03-12T03:17:01.720 INFO:tasks.cephadm:Adding mon.smithi177 on smithi177 2024-03-12T03:17:01.720 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch apply mon '2;smithi118:172.21.15.118=smithi118;smithi177:172.21.15.177=smithi177' 2024-03-12T03:17:01.925 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:01.970 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:02.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:01 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/980716675' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:17:02.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:01 smithi118 ceph-mon[29160]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:17:02.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:01 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:02 smithi118 ceph-mon[29160]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.374+0000 7fbc17f61700 1 -- 172.21.15.177:0/2761911408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 msgr2=0x7fbc10100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.374+0000 7fbc17f61700 1 --2- 172.21.15.177:0/2761911408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10100b10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009910 tx=0x7fbc0c009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.374+0000 7fbc17f61700 1 -- 172.21.15.177:0/2761911408 shutdown_connections 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.374+0000 7fbc17f61700 1 --2- 172.21.15.177:0/2761911408 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10100b10 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.374+0000 7fbc17f61700 1 -- 172.21.15.177:0/2761911408 >> 172.21.15.177:0/2761911408 conn(0x7fbc100fbcb0 msgr2=0x7fbc100fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.375+0000 7fbc17f61700 1 -- 172.21.15.177:0/2761911408 shutdown_connections 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.375+0000 7fbc17f61700 1 -- 172.21.15.177:0/2761911408 wait complete. 2024-03-12T03:17:03.379 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.375+0000 7fbc17f61700 1 Processor -- start 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.375+0000 7fbc17f61700 1 -- start start 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.375+0000 7fbc17f61700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10195b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.375+0000 7fbc17f61700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc101960a0 con 0x7fbc101006f0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.375+0000 7fbc15cfd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10195b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc15cfd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10195b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:52274/0 (socket says 172.21.15.177:52274) 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc15cfd700 1 -- 172.21.15.177:0/9275698 learned_addr learned my addr 172.21.15.177:0/9275698 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc15cfd700 1 -- 172.21.15.177:0/9275698 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0095f0 con 0x7fbc101006f0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc15cfd700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10195b60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c006010 tx=0x7fbc0c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc06ffd700 1 -- 172.21.15.177:0/9275698 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc0c01d070 con 0x7fbc101006f0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc101962a0 con 0x7fbc101006f0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc06ffd700 1 -- 172.21.15.177:0/9275698 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbc0c005dc0 con 0x7fbc101006f0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc06ffd700 1 -- 172.21.15.177:0/9275698 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc0c022e90 con 0x7fbc101006f0 2024-03-12T03:17:03.380 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.376+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc10196740 con 0x7fbc101006f0 2024-03-12T03:17:03.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.377+0000 7fbc06ffd700 1 -- 172.21.15.177:0/9275698 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7fbc0c00f4f0 con 0x7fbc101006f0 2024-03-12T03:17:03.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.377+0000 7fbc06ffd700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fbbfc038530 0x7fbbfc03a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:03.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.377+0000 7fbc06ffd700 1 -- 172.21.15.177:0/9275698 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbc0c04c5c0 con 0x7fbc101006f0 2024-03-12T03:17:03.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.377+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1018f520 con 0x7fbc101006f0 2024-03-12T03:17:03.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.377+0000 7fbc154fc700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fbbfc038530 0x7fbbfc03a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:03.381 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.378+0000 7fbc154fc700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fbbfc038530 0x7fbbfc03a9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbc00006fd0 tx=0x7fbc00006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:03.384 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.382+0000 7fbc06ffd700 1 -- 172.21.15.177:0/9275698 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fbc0c027070 con 0x7fbc101006f0 2024-03-12T03:17:03.587 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.585+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 --> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi118:172.21.15.118=smithi118;smithi177:172.21.15.177=smithi177", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc100611f0 con 0x7fbbfc038530 2024-03-12T03:17:03.593 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.591+0000 7fbc06ffd700 1 -- 172.21.15.177:0/9275698 <== mgr.14164 v2:172.21.15.118:6800/180481365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbc100611f0 con 0x7fbbfc038530 2024-03-12T03:17:03.593 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-03-12T03:17:03.595 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fbbfc038530 msgr2=0x7fbbfc03a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:03.595 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fbbfc038530 0x7fbbfc03a9f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbc00006fd0 tx=0x7fbc00006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:03.595 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 msgr2=0x7fbc10195b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:03.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10195b60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c006010 tx=0x7fbc0c009f80 comp rx=0 tx=0).stop 2024-03-12T03:17:03.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 shutdown_connections 2024-03-12T03:17:03.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fbbfc038530 0x7fbbfc03a9f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:03.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 --2- 172.21.15.177:0/9275698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc101006f0 0x7fbc10195b60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:03.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.592+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 >> 172.21.15.177:0/9275698 conn(0x7fbc100fbcb0 msgr2=0x7fbc100fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:03.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.593+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 shutdown_connections 2024-03-12T03:17:03.596 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:03.593+0000 7fbc17f61700 1 -- 172.21.15.177:0/9275698 wait complete. 2024-03-12T03:17:04.118 DEBUG:teuthology.orchestra.run.smithi177:mon.smithi177> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi177.service 2024-03-12T03:17:04.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:04.121 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:04.350 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:04.397 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:04.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:04 smithi118 ceph-mon[29160]: from='client.14197 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi118:172.21.15.118=smithi118;smithi177:172.21.15.177=smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:17:04.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:04 smithi118 ceph-mon[29160]: Saving service mon spec with placement smithi118:172.21.15.118=smithi118;smithi177:172.21.15.177=smithi177;count:2 2024-03-12T03:17:04.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:04 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:04.870 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:04 smithi118 ceph-mon[29160]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:05.723 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.720+0000 7f55ecb06700 1 -- 172.21.15.177:0/3837306062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 msgr2=0x7f55e8100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:05.723 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.720+0000 7f55ecb06700 1 --2- 172.21.15.177:0/3837306062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8100b00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f55d8009910 tx=0x7f55d8009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:05.723 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.720+0000 7f55ecb06700 1 -- 172.21.15.177:0/3837306062 shutdown_connections 2024-03-12T03:17:05.723 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.720+0000 7f55ecb06700 1 --2- 172.21.15.177:0/3837306062 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8100b00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:05.723 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.720+0000 7f55ecb06700 1 -- 172.21.15.177:0/3837306062 >> 172.21.15.177:0/3837306062 conn(0x7f55e80fbcc0 msgr2=0x7f55e80fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:05.723 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.720+0000 7f55ecb06700 1 -- 172.21.15.177:0/3837306062 shutdown_connections 2024-03-12T03:17:05.723 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.720+0000 7f55ecb06700 1 -- 172.21.15.177:0/3837306062 wait complete. 2024-03-12T03:17:05.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55ecb06700 1 Processor -- start 2024-03-12T03:17:05.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55ecb06700 1 -- start start 2024-03-12T03:17:05.725 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55ecb06700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8074a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55ecb06700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55e8074f60 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55e659c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8074a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55e659c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8074a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:52288/0 (socket says 172.21.15.177:52288) 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55e659c700 1 -- 172.21.15.177:0/58711090 learned_addr learned my addr 172.21.15.177:0/58711090 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.721+0000 7f55e659c700 1 -- 172.21.15.177:0/58711090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d80095f0 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.722+0000 7f55e659c700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8074a20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f55d8006010 tx=0x7f55d80050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.722+0000 7f55d77fe700 1 -- 172.21.15.177:0/58711090 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55d801d070 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.722+0000 7f55d77fe700 1 -- 172.21.15.177:0/58711090 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55d800be20 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.722+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55e80731a0 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.722+0000 7f55d77fe700 1 -- 172.21.15.177:0/58711090 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55d8022de0 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.722+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55e8073640 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.723+0000 7f55d77fe700 1 -- 172.21.15.177:0/58711090 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f55d800b960 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.723+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55e8106e40 con 0x7f55e81006e0 2024-03-12T03:17:05.726 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.723+0000 7f55d77fe700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f55d0038580 0x7f55d003aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:05.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.723+0000 7f55d77fe700 1 -- 172.21.15.177:0/58711090 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f55d804c6a0 con 0x7f55e81006e0 2024-03-12T03:17:05.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.723+0000 7f55e5d9b700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f55d0038580 0x7f55d003aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:05.727 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.723+0000 7f55e5d9b700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f55d0038580 0x7f55d003aa40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55dc006fd0 tx=0x7f55dc006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:05.730 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:05.728+0000 7f55d77fe700 1 -- 172.21.15.177:0/58711090 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f55d80226c0 con 0x7f55e81006e0 2024-03-12T03:17:06.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.002+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55e80623e0 con 0x7f55e81006e0 2024-03-12T03:17:06.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.002+0000 7f55d77fe700 1 -- 172.21.15.177:0/58711090 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f55d8027030 con 0x7f55e81006e0 2024-03-12T03:17:06.005 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:06.005 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:06.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f55d0038580 msgr2=0x7f55d003aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:06.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f55d0038580 0x7f55d003aa40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f55dc006fd0 tx=0x7f55dc006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:06.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 msgr2=0x7f55e8074a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:06.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8074a20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f55d8006010 tx=0x7f55d80050b0 comp rx=0 tx=0).stop 2024-03-12T03:17:06.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 shutdown_connections 2024-03-12T03:17:06.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f55d0038580 0x7f55d003aa40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:06.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 --2- 172.21.15.177:0/58711090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f55e81006e0 0x7f55e8074a20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:06.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 >> 172.21.15.177:0/58711090 conn(0x7f55e80fbcc0 msgr2=0x7f55e80fc760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:06.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 shutdown_connections 2024-03-12T03:17:06.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:06.004+0000 7f55ecb06700 1 -- 172.21.15.177:0/58711090 wait complete. 2024-03-12T03:17:06.008 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:07.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:06 smithi118 ceph-mon[29160]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:07.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:06 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/58711090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:07.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:07.478 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:07.671 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:07.717 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:08.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:08 smithi118 ceph-mon[29160]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:09.003 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.996+0000 7f3036cfe700 1 -- 172.21.15.177:0/3951535075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 msgr2=0x7f30300ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:09.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.996+0000 7f3036cfe700 1 --2- 172.21.15.177:0/3951535075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300ffa10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3024009910 tx=0x7f3024009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:09.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.996+0000 7f3036cfe700 1 -- 172.21.15.177:0/3951535075 shutdown_connections 2024-03-12T03:17:09.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.996+0000 7f3036cfe700 1 --2- 172.21.15.177:0/3951535075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300ffa10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:09.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.996+0000 7f3036cfe700 1 -- 172.21.15.177:0/3951535075 >> 172.21.15.177:0/3951535075 conn(0x7f30300fab70 msgr2=0x7f30300fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:09.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.996+0000 7f3036cfe700 1 -- 172.21.15.177:0/3951535075 shutdown_connections 2024-03-12T03:17:09.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.996+0000 7f3036cfe700 1 -- 172.21.15.177:0/3951535075 wait complete. 2024-03-12T03:17:09.004 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.997+0000 7f3036cfe700 1 Processor -- start 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.997+0000 7f3036cfe700 1 -- start start 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.997+0000 7f3036cfe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300fca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.997+0000 7f3036cfe700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30300fcf40 con 0x7f30300ff5f0 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.997+0000 7f3034a9a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300fca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.997+0000 7f3034a9a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300fca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:52304/0 (socket says 172.21.15.177:52304) 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.997+0000 7f3034a9a700 1 -- 172.21.15.177:0/1047818479 learned_addr learned my addr 172.21.15.177:0/1047818479 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.998+0000 7f3034a9a700 1 -- 172.21.15.177:0/1047818479 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30240095f0 con 0x7f30300ff5f0 2024-03-12T03:17:09.005 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.998+0000 7f3034a9a700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300fca00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3024006010 tx=0x7f30240050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:09.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.998+0000 7f302dffb700 1 -- 172.21.15.177:0/1047818479 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302401d070 con 0x7f30300ff5f0 2024-03-12T03:17:09.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.998+0000 7f302dffb700 1 -- 172.21.15.177:0/1047818479 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f302400be20 con 0x7f30300ff5f0 2024-03-12T03:17:09.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.998+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30300fb180 con 0x7f30300ff5f0 2024-03-12T03:17:09.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.998+0000 7f302dffb700 1 -- 172.21.15.177:0/1047818479 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3024022de0 con 0x7f30300ff5f0 2024-03-12T03:17:09.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.998+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30300fb620 con 0x7f30300ff5f0 2024-03-12T03:17:09.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.999+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3030105d50 con 0x7f30300ff5f0 2024-03-12T03:17:09.006 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.999+0000 7f302dffb700 1 -- 172.21.15.177:0/1047818479 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f30240226c0 con 0x7f30300ff5f0 2024-03-12T03:17:09.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.999+0000 7f302dffb700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f30200385f0 0x7f302003aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:09.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.999+0000 7f302dffb700 1 -- 172.21.15.177:0/1047818479 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f302404c7e0 con 0x7f30300ff5f0 2024-03-12T03:17:09.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:08.999+0000 7f302ffff700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f30200385f0 0x7f302003aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:09.007 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.000+0000 7f302ffff700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f30200385f0 0x7f302003aab0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f301c006fd0 tx=0x7f301c006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:09.008 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.004+0000 7f302dffb700 1 -- 172.21.15.177:0/1047818479 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f302402ba00 con 0x7f30300ff5f0 2024-03-12T03:17:09.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.272+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f303018f640 con 0x7f30300ff5f0 2024-03-12T03:17:09.275 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.273+0000 7f302dffb700 1 -- 172.21.15.177:0/1047818479 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f3024027030 con 0x7f30300ff5f0 2024-03-12T03:17:09.275 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:09.275 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:09.277 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f30200385f0 msgr2=0x7f302003aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f30200385f0 0x7f302003aab0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f301c006fd0 tx=0x7f301c006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 msgr2=0x7f30300fca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300fca00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3024006010 tx=0x7f30240050b0 comp rx=0 tx=0).stop 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 shutdown_connections 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f30200385f0 0x7f302003aab0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 --2- 172.21.15.177:0/1047818479 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30300ff5f0 0x7f30300fca00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 >> 172.21.15.177:0/1047818479 conn(0x7f30300fab70 msgr2=0x7f30300682b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 shutdown_connections 2024-03-12T03:17:09.278 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:09.275+0000 7f3036cfe700 1 -- 172.21.15.177:0/1047818479 wait complete. 2024-03-12T03:17:09.279 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:17:10.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:17:10.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:17:10.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:10.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: Deploying daemon grafana.smithi118 on smithi118 2024-03-12T03:17:10.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1047818479' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:10.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:09 smithi118 ceph-mon[29160]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:10.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:10.782 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:10.984 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:11.030 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:12.477 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 -- 172.21.15.177:0/2657698257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 msgr2=0x7f1de00fedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:12.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 --2- 172.21.15.177:0/2657698257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de00fedf0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009910 tx=0x7f1dd0009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:12.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 -- 172.21.15.177:0/2657698257 shutdown_connections 2024-03-12T03:17:12.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 --2- 172.21.15.177:0/2657698257 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de00fedf0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:12.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 -- 172.21.15.177:0/2657698257 >> 172.21.15.177:0/2657698257 conn(0x7f1de00fa310 msgr2=0x7f1de00fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:12.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 -- 172.21.15.177:0/2657698257 shutdown_connections 2024-03-12T03:17:12.478 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 -- 172.21.15.177:0/2657698257 wait complete. 2024-03-12T03:17:12.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 Processor -- start 2024-03-12T03:17:12.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 -- start start 2024-03-12T03:17:12.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de0195bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:12.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1de5c96700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de0196100 con 0x7f1de00fe9d0 2024-03-12T03:17:12.479 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1ddf7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de0195bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1ddf7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de0195bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:41138/0 (socket says 172.21.15.177:41138) 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1ddf7fe700 1 -- 172.21.15.177:0/3624905302 learned_addr learned my addr 172.21.15.177:0/3624905302 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1ddf7fe700 1 -- 172.21.15.177:0/3624905302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00095f0 con 0x7f1de00fe9d0 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.476+0000 7f1ddf7fe700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de0195bc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0006010 tx=0x7f1dd0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.477+0000 7f1ddcff9700 1 -- 172.21.15.177:0/3624905302 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dd001d070 con 0x7f1de00fe9d0 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.477+0000 7f1ddcff9700 1 -- 172.21.15.177:0/3624905302 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1dd0005dc0 con 0x7f1de00fe9d0 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.477+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de0196300 con 0x7f1de00fe9d0 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.477+0000 7f1ddcff9700 1 -- 172.21.15.177:0/3624905302 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dd0022e90 con 0x7f1de00fe9d0 2024-03-12T03:17:12.480 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.477+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de01967a0 con 0x7f1de00fe9d0 2024-03-12T03:17:12.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.477+0000 7f1ddcff9700 1 -- 172.21.15.177:0/3624905302 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f1dd000f4f0 con 0x7f1de00fe9d0 2024-03-12T03:17:12.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.478+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de018f550 con 0x7f1de00fe9d0 2024-03-12T03:17:12.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.478+0000 7f1ddcff9700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1dc8038530 0x7f1dc803a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:12.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.478+0000 7f1ddcff9700 1 -- 172.21.15.177:0/3624905302 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1dd004c430 con 0x7f1de00fe9d0 2024-03-12T03:17:12.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.478+0000 7f1ddeffd700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1dc8038530 0x7f1dc803a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:12.481 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.478+0000 7f1ddeffd700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1dc8038530 0x7f1dc803a9f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4006fd0 tx=0x7f1dd4006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:12.485 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.483+0000 7f1ddcff9700 1 -- 172.21.15.177:0/3624905302 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1dd0017440 con 0x7f1de00fe9d0 2024-03-12T03:17:12.753 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.751+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1de00ff500 con 0x7f1de00fe9d0 2024-03-12T03:17:12.754 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.752+0000 7f1ddcff9700 1 -- 172.21.15.177:0/3624905302 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f1de00ff500 con 0x7f1de00fe9d0 2024-03-12T03:17:12.754 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:12.754 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:12.755 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1dc8038530 msgr2=0x7f1dc803a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1dc8038530 0x7f1dc803a9f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4006fd0 tx=0x7f1dd4006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 msgr2=0x7f1de0195bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de0195bc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0006010 tx=0x7f1dd0009f80 comp rx=0 tx=0).stop 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 shutdown_connections 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f1dc8038530 0x7f1dc803a9f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 --2- 172.21.15.177:0/3624905302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1de00fe9d0 0x7f1de0195bc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 >> 172.21.15.177:0/3624905302 conn(0x7f1de00fa310 msgr2=0x7f1de00fada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 shutdown_connections 2024-03-12T03:17:12.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:12.754+0000 7f1de5c96700 1 -- 172.21.15.177:0/3624905302 wait complete. 2024-03-12T03:17:12.757 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:13.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:12 smithi118 ceph-mon[29160]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:13.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:12 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:14.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:13 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/3624905302' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:14.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:14.352 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:14.557 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:14.603 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:15.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:14 smithi118 ceph-mon[29160]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.102+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/3924854023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 msgr2=0x7f4fb8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.102+0000 7f4fbe0ba700 1 --2- 172.21.15.177:0/3924854023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8100b10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009910 tx=0x7f4fa8009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.102+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/3924854023 shutdown_connections 2024-03-12T03:17:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.102+0000 7f4fbe0ba700 1 --2- 172.21.15.177:0/3924854023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8100b10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.102+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/3924854023 >> 172.21.15.177:0/3924854023 conn(0x7f4fb80fbcb0 msgr2=0x7f4fb80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.102+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/3924854023 shutdown_connections 2024-03-12T03:17:16.105 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.102+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/3924854023 wait complete. 2024-03-12T03:17:16.106 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.103+0000 7f4fbe0ba700 1 Processor -- start 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.103+0000 7f4fbe0ba700 1 -- start start 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.103+0000 7f4fbe0ba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.103+0000 7f4fbe0ba700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb81960b0 con 0x7f4fb81006f0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.103+0000 7f4fb77fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.103+0000 7f4fb77fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8195b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:41156/0 (socket says 172.21.15.177:41156) 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.103+0000 7f4fb77fe700 1 -- 172.21.15.177:0/1181381192 learned_addr learned my addr 172.21.15.177:0/1181381192 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.104+0000 7f4fb77fe700 1 -- 172.21.15.177:0/1181381192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa80095f0 con 0x7f4fb81006f0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.104+0000 7f4fb77fe700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8195b70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8006010 tx=0x7f4fa8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.104+0000 7f4fb4ff9700 1 -- 172.21.15.177:0/1181381192 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fa801d070 con 0x7f4fb81006f0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.104+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb81962b0 con 0x7f4fb81006f0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.104+0000 7f4fb4ff9700 1 -- 172.21.15.177:0/1181381192 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4fa8005dc0 con 0x7f4fb81006f0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.104+0000 7f4fb4ff9700 1 -- 172.21.15.177:0/1181381192 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fa8022e90 con 0x7f4fb81006f0 2024-03-12T03:17:16.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.104+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb8196750 con 0x7f4fb81006f0 2024-03-12T03:17:16.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.105+0000 7f4fb4ff9700 1 -- 172.21.15.177:0/1181381192 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f4fa8022470 con 0x7f4fb81006f0 2024-03-12T03:17:16.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.105+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb818f520 con 0x7f4fb81006f0 2024-03-12T03:17:16.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.105+0000 7f4fb4ff9700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4fa0038530 0x7f4fa003a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:16.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.105+0000 7f4fb4ff9700 1 -- 172.21.15.177:0/1181381192 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4fa804c700 con 0x7f4fb81006f0 2024-03-12T03:17:16.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.105+0000 7f4fb6ffd700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4fa0038530 0x7f4fa003a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:16.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.106+0000 7f4fb6ffd700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4fa0038530 0x7f4fa003a9f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4fac006fd0 tx=0x7f4fac006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:16.115 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.112+0000 7f4fb4ff9700 1 -- 172.21.15.177:0/1181381192 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4fa800b960 con 0x7f4fb81006f0 2024-03-12T03:17:16.387 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.385+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fb80623e0 con 0x7f4fb81006f0 2024-03-12T03:17:16.388 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.386+0000 7f4fb4ff9700 1 -- 172.21.15.177:0/1181381192 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f4fa8027020 con 0x7f4fb81006f0 2024-03-12T03:17:16.388 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:16.388 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4fa0038530 msgr2=0x7f4fa003a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4fa0038530 0x7f4fa003a9f0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4fac006fd0 tx=0x7f4fac006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 msgr2=0x7f4fb8195b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8195b70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8006010 tx=0x7f4fa8009f80 comp rx=0 tx=0).stop 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 shutdown_connections 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f4fa0038530 0x7f4fa003a9f0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 --2- 172.21.15.177:0/1181381192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4fb81006f0 0x7f4fb8195b70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 >> 172.21.15.177:0/1181381192 conn(0x7f4fb80fbcb0 msgr2=0x7f4fb80fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:16.391 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 shutdown_connections 2024-03-12T03:17:16.392 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:16.388+0000 7f4fbe0ba700 1 -- 172.21.15.177:0/1181381192 wait complete. 2024-03-12T03:17:16.392 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:17.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:16 smithi118 ceph-mon[29160]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:17.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:16 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1181381192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:17.973 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:17.973 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:18.175 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:18.221 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:19.273 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:18 smithi118 ceph-mon[29160]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:19.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.629+0000 7f060251f700 1 -- 172.21.15.177:0/1676102787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 msgr2=0x7f05fc0ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:19.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.629+0000 7f060251f700 1 --2- 172.21.15.177:0/1676102787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0ffa10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f05f0009910 tx=0x7f05f0009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:19.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.629+0000 7f060251f700 1 -- 172.21.15.177:0/1676102787 shutdown_connections 2024-03-12T03:17:19.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.629+0000 7f060251f700 1 --2- 172.21.15.177:0/1676102787 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0ffa10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:19.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.629+0000 7f060251f700 1 -- 172.21.15.177:0/1676102787 >> 172.21.15.177:0/1676102787 conn(0x7f05fc0fab90 msgr2=0x7f05fc0fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:19.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.629+0000 7f060251f700 1 -- 172.21.15.177:0/1676102787 shutdown_connections 2024-03-12T03:17:19.632 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.629+0000 7f060251f700 1 -- 172.21.15.177:0/1676102787 wait complete. 2024-03-12T03:17:19.633 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.630+0000 7f060251f700 1 Processor -- start 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.630+0000 7f060251f700 1 -- start start 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.630+0000 7f060251f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.630+0000 7f060251f700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc0699f0 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.630+0000 7f05fbfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.630+0000 7f05fbfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0694b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:41172/0 (socket says 172.21.15.177:41172) 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.630+0000 7f05fbfff700 1 -- 172.21.15.177:0/3966563205 learned_addr learned my addr 172.21.15.177:0/3966563205 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.631+0000 7f05fbfff700 1 -- 172.21.15.177:0/3966563205 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f00095f0 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.631+0000 7f05fbfff700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0694b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f05f0006010 tx=0x7f05f0009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.631+0000 7f05f97fa700 1 -- 172.21.15.177:0/3966563205 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f001d070 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.631+0000 7f05f97fa700 1 -- 172.21.15.177:0/3966563205 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05f0005dc0 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.631+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc1966e0 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.631+0000 7f05f97fa700 1 -- 172.21.15.177:0/3966563205 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f05f0022e90 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.631+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc19af40 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.635 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.632+0000 7f05f97fa700 1 -- 172.21.15.177:0/3966563205 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f05f000b960 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.635 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.632+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc18e420 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.635 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.632+0000 7f05f97fa700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f05e4038530 0x7f05e403a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:19.635 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.632+0000 7f05f97fa700 1 -- 172.21.15.177:0/3966563205 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f05f004c5b0 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.632+0000 7f05fb7fe700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f05e4038530 0x7f05e403a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:19.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.633+0000 7f05fb7fe700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f05e4038530 0x7f05e403a9f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f05ec006fd0 tx=0x7f05ec006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:19.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.641+0000 7f05f97fa700 1 -- 172.21.15.177:0/3966563205 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f05f000bd00 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.914 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.912+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f05fc0623e0 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.915 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.913+0000 7f05f97fa700 1 -- 172.21.15.177:0/3966563205 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f05f0027020 con 0x7f05fc0ff5f0 2024-03-12T03:17:19.915 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:19.915 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:19.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.914+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f05e4038530 msgr2=0x7f05e403a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:19.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.914+0000 7f060251f700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f05e4038530 0x7f05e403a9f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f05ec006fd0 tx=0x7f05ec006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:19.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 msgr2=0x7f05fc0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:19.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0694b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f05f0006010 tx=0x7f05f0009f80 comp rx=0 tx=0).stop 2024-03-12T03:17:19.917 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 shutdown_connections 2024-03-12T03:17:19.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f05e4038530 0x7f05e403a9f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:19.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 --2- 172.21.15.177:0/3966563205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff5f0 0x7f05fc0694b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:19.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 >> 172.21.15.177:0/3966563205 conn(0x7f05fc0fab90 msgr2=0x7f05fc0fb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:19.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 shutdown_connections 2024-03-12T03:17:19.918 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:19.915+0000 7f060251f700 1 -- 172.21.15.177:0/3966563205 wait complete. 2024-03-12T03:17:19.918 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:21.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:20 smithi118 ceph-mon[29160]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:21.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:20 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/3966563205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:21.383 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:21.383 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:21.586 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:21.631 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.952+0000 7ffb8db4b700 1 -- 172.21.15.177:0/728262572 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 msgr2=0x7ffb880ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.952+0000 7ffb8db4b700 1 --2- 172.21.15.177:0/728262572 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880ffa10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffb78009910 tx=0x7ffb78009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.952+0000 7ffb8db4b700 1 -- 172.21.15.177:0/728262572 shutdown_connections 2024-03-12T03:17:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.952+0000 7ffb8db4b700 1 --2- 172.21.15.177:0/728262572 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880ffa10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.952+0000 7ffb8db4b700 1 -- 172.21.15.177:0/728262572 >> 172.21.15.177:0/728262572 conn(0x7ffb880fab70 msgr2=0x7ffb880fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.952+0000 7ffb8db4b700 1 -- 172.21.15.177:0/728262572 shutdown_connections 2024-03-12T03:17:22.955 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.952+0000 7ffb8db4b700 1 -- 172.21.15.177:0/728262572 wait complete. 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.953+0000 7ffb8db4b700 1 Processor -- start 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.953+0000 7ffb8db4b700 1 -- start start 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.953+0000 7ffb8db4b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880688c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.953+0000 7ffb8db4b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb88068e00 con 0x7ffb880ff5f0 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.953+0000 7ffb877fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.953+0000 7ffb877fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880688c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:45618/0 (socket says 172.21.15.177:45618) 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.953+0000 7ffb877fe700 1 -- 172.21.15.177:0/3298050409 learned_addr learned my addr 172.21.15.177:0/3298050409 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:22.957 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.954+0000 7ffb877fe700 1 -- 172.21.15.177:0/3298050409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780095f0 con 0x7ffb880ff5f0 2024-03-12T03:17:22.958 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.954+0000 7ffb877fe700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880688c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffb78006010 tx=0x7ffb78009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:22.958 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.954+0000 7ffb857fa700 1 -- 172.21.15.177:0/3298050409 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb7801d070 con 0x7ffb880ff5f0 2024-03-12T03:17:22.958 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.954+0000 7ffb857fa700 1 -- 172.21.15.177:0/3298050409 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb78005dc0 con 0x7ffb880ff5f0 2024-03-12T03:17:22.958 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.954+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb881988c0 con 0x7ffb880ff5f0 2024-03-12T03:17:22.958 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.954+0000 7ffb857fa700 1 -- 172.21.15.177:0/3298050409 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb78022e90 con 0x7ffb880ff5f0 2024-03-12T03:17:22.958 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.954+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb88198d60 con 0x7ffb880ff5f0 2024-03-12T03:17:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.955+0000 7ffb857fa700 1 -- 172.21.15.177:0/3298050409 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7ffb7800b960 con 0x7ffb880ff5f0 2024-03-12T03:17:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.955+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8818e420 con 0x7ffb880ff5f0 2024-03-12T03:17:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.955+0000 7ffb857fa700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7ffb74038530 0x7ffb7403a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.955+0000 7ffb857fa700 1 -- 172.21.15.177:0/3298050409 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffb7804c520 con 0x7ffb880ff5f0 2024-03-12T03:17:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.955+0000 7ffb7ffff700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7ffb74038530 0x7ffb7403a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:22.959 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.956+0000 7ffb7ffff700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7ffb74038530 0x7ffb7403a9f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffb70006fd0 tx=0x7ffb70006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:22.962 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:22.960+0000 7ffb857fa700 1 -- 172.21.15.177:0/3298050409 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffb78022750 con 0x7ffb880ff5f0 2024-03-12T03:17:23.213 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:22 smithi118 ceph-mon[29160]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:23.231 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.229+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb8802d110 con 0x7ffb880ff5f0 2024-03-12T03:17:23.233 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.230+0000 7ffb857fa700 1 -- 172.21.15.177:0/3298050409 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7ffb78027020 con 0x7ffb880ff5f0 2024-03-12T03:17:23.233 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:23.233 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:23.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7ffb74038530 msgr2=0x7ffb7403a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:23.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7ffb74038530 0x7ffb7403a9f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffb70006fd0 tx=0x7ffb70006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:23.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 msgr2=0x7ffb880688c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:23.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880688c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ffb78006010 tx=0x7ffb78009f80 comp rx=0 tx=0).stop 2024-03-12T03:17:23.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 shutdown_connections 2024-03-12T03:17:23.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7ffb74038530 0x7ffb7403a9f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:23.235 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 --2- 172.21.15.177:0/3298050409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb880ff5f0 0x7ffb880688c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:23.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 >> 172.21.15.177:0/3298050409 conn(0x7ffb880fab70 msgr2=0x7ffb880fb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:23.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 shutdown_connections 2024-03-12T03:17:23.236 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:23.232+0000 7ffb8db4b700 1 -- 172.21.15.177:0/3298050409 wait complete. 2024-03-12T03:17:23.236 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:24.017 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/3298050409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.018 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:23 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:24.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:24.792 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:24.994 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:25.040 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:25.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:24 smithi118 ceph-mon[29160]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:26.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:25 smithi118 ceph-mon[29160]: Deploying daemon prometheus.smithi118 on smithi118 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 -- 172.21.15.177:0/1446664025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 msgr2=0x7f5e70103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 --2- 172.21.15.177:0/1446664025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e70103590 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009910 tx=0x7f5e6c009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 -- 172.21.15.177:0/1446664025 shutdown_connections 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 --2- 172.21.15.177:0/1446664025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e70103590 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 -- 172.21.15.177:0/1446664025 >> 172.21.15.177:0/1446664025 conn(0x7f5e700fabb0 msgr2=0x7f5e700fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 -- 172.21.15.177:0/1446664025 shutdown_connections 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 -- 172.21.15.177:0/1446664025 wait complete. 2024-03-12T03:17:26.460 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 Processor -- start 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.456+0000 7f5e78697700 1 -- start start 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e78697700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e701009d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e78697700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70100f10 con 0x7f5e701011a0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e76433700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e701009d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e76433700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e701009d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:45638/0 (socket says 172.21.15.177:45638) 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e76433700 1 -- 172.21.15.177:0/2972136214 learned_addr learned my addr 172.21.15.177:0/2972136214 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e76433700 1 -- 172.21.15.177:0/2972136214 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c0095f0 con 0x7f5e701011a0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e76433700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e701009d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c005b40 tx=0x7f5e6c005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e677fe700 1 -- 172.21.15.177:0/2972136214 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e6c01d070 con 0x7f5e701011a0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e700ff150 con 0x7f5e701011a0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e677fe700 1 -- 172.21.15.177:0/2972136214 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e6c005dc0 con 0x7f5e701011a0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e677fe700 1 -- 172.21.15.177:0/2972136214 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e6c022e00 con 0x7f5e701011a0 2024-03-12T03:17:26.461 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.457+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e700ff5f0 con 0x7f5e701011a0 2024-03-12T03:17:26.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.458+0000 7f5e677fe700 1 -- 172.21.15.177:0/2972136214 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f5e6c00b960 con 0x7f5e701011a0 2024-03-12T03:17:26.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.458+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70106e70 con 0x7f5e701011a0 2024-03-12T03:17:26.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.458+0000 7f5e677fe700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f5e5c038530 0x7f5e5c03a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:26.462 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.458+0000 7f5e677fe700 1 -- 172.21.15.177:0/2972136214 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5e6c04c640 con 0x7f5e701011a0 2024-03-12T03:17:26.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.458+0000 7f5e75c32700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f5e5c038530 0x7f5e5c03a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:26.463 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.459+0000 7f5e75c32700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f5e5c038530 0x7f5e5c03a9f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e60006fd0 tx=0x7f5e60006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:26.466 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.464+0000 7f5e677fe700 1 -- 172.21.15.177:0/2972136214 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5e6c022690 con 0x7f5e701011a0 2024-03-12T03:17:26.738 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.735+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e700623e0 con 0x7f5e701011a0 2024-03-12T03:17:26.739 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.736+0000 7f5e677fe700 1 -- 172.21.15.177:0/2972136214 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f5e6c027030 con 0x7f5e701011a0 2024-03-12T03:17:26.739 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:26.739 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f5e5c038530 msgr2=0x7f5e5c03a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f5e5c038530 0x7f5e5c03a9f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e60006fd0 tx=0x7f5e60006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 msgr2=0x7f5e701009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e701009d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c005b40 tx=0x7f5e6c005000 comp rx=0 tx=0).stop 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 shutdown_connections 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f5e5c038530 0x7f5e5c03a9f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 --2- 172.21.15.177:0/2972136214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5e701011a0 0x7f5e701009d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 >> 172.21.15.177:0/2972136214 conn(0x7f5e700fabb0 msgr2=0x7f5e700fb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 shutdown_connections 2024-03-12T03:17:26.741 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:26.738+0000 7f5e78697700 1 -- 172.21.15.177:0/2972136214 wait complete. 2024-03-12T03:17:26.742 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:27.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:26 smithi118 ceph-mon[29160]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:27.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:26 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:27.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:26 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/2972136214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:28.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:28.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:28.463 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:28.509 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:29.287 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:28 smithi118 ceph-mon[29160]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:30.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 -- 172.21.15.177:0/1142177087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 msgr2=0x7f62bc0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:30.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 --2- 172.21.15.177:0/1142177087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc0ffa30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009910 tx=0x7f62ac009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:30.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 -- 172.21.15.177:0/1142177087 shutdown_connections 2024-03-12T03:17:30.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 --2- 172.21.15.177:0/1142177087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc0ffa30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:30.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 -- 172.21.15.177:0/1142177087 >> 172.21.15.177:0/1142177087 conn(0x7f62bc0fabb0 msgr2=0x7f62bc0fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:30.142 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 -- 172.21.15.177:0/1142177087 shutdown_connections 2024-03-12T03:17:30.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 -- 172.21.15.177:0/1142177087 wait complete. 2024-03-12T03:17:30.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.139+0000 7f62c1d4f700 1 Processor -- start 2024-03-12T03:17:30.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.140+0000 7f62c1d4f700 1 -- start start 2024-03-12T03:17:30.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.140+0000 7f62c1d4f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc195c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:30.143 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.140+0000 7f62c1d4f700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62bc196150 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.140+0000 7f62bb7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc195c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.140+0000 7f62bb7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc195c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:45660/0 (socket says 172.21.15.177:45660) 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.140+0000 7f62bb7fe700 1 -- 172.21.15.177:0/171509715 learned_addr learned my addr 172.21.15.177:0/171509715 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.140+0000 7f62bb7fe700 1 -- 172.21.15.177:0/171509715 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac0095f0 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.141+0000 7f62bb7fe700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc195c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f62ac004f40 tx=0x7f62ac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.141+0000 7f62b8ff9700 1 -- 172.21.15.177:0/171509715 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac01d070 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.141+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62bc196350 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.141+0000 7f62b8ff9700 1 -- 172.21.15.177:0/171509715 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62ac005dc0 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.141+0000 7f62b8ff9700 1 -- 172.21.15.177:0/171509715 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62ac022e40 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.141+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62bc1967f0 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.142+0000 7f62b8ff9700 1 -- 172.21.15.177:0/171509715 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f62ac00b960 con 0x7f62bc0ff610 2024-03-12T03:17:30.145 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.142+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62bc0623e0 con 0x7f62bc0ff610 2024-03-12T03:17:30.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.142+0000 7f62b8ff9700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f62a4038530 0x7f62a403a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:30.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.142+0000 7f62b8ff9700 1 -- 172.21.15.177:0/171509715 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f62ac04c590 con 0x7f62bc0ff610 2024-03-12T03:17:30.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.142+0000 7f62baffd700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f62a4038530 0x7f62a403a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:30.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.142+0000 7f62baffd700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f62a4038530 0x7f62a403a9f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f62b0006fd0 tx=0x7f62b0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:30.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.147+0000 7f62b8ff9700 1 -- 172.21.15.177:0/171509715 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f62ac0226c0 con 0x7f62bc0ff610 2024-03-12T03:17:30.422 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.420+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62bc199080 con 0x7f62bc0ff610 2024-03-12T03:17:30.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.420+0000 7f62b8ff9700 1 -- 172.21.15.177:0/171509715 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f62ac027030 con 0x7f62bc0ff610 2024-03-12T03:17:30.423 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:30.423 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:30.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f62a4038530 msgr2=0x7f62a403a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:30.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f62a4038530 0x7f62a403a9f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f62b0006fd0 tx=0x7f62b0006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:30.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 msgr2=0x7f62bc195c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc195c10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f62ac004f40 tx=0x7f62ac004740 comp rx=0 tx=0).stop 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 shutdown_connections 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f62a4038530 0x7f62a403a9f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 --2- 172.21.15.177:0/171509715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f62bc0ff610 0x7f62bc195c10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 >> 172.21.15.177:0/171509715 conn(0x7f62bc0fabb0 msgr2=0x7f62bc068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 shutdown_connections 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:30.423+0000 7f62c1d4f700 1 -- 172.21.15.177:0/171509715 wait complete. 2024-03-12T03:17:30.426 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:31.295 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:30 smithi118 ceph-mon[29160]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:31.295 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:30 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/171509715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:31.995 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:31.995 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:32.196 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:32.242 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:33.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:32 smithi118 ceph-mon[29160]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:33.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.561+0000 7f7efd428700 1 -- 172.21.15.177:0/3678084993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 msgr2=0x7f7ef8103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:33.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.561+0000 7f7efd428700 1 --2- 172.21.15.177:0/3678084993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8103570 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009910 tx=0x7f7ee8009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:33.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.561+0000 7f7efd428700 1 -- 172.21.15.177:0/3678084993 shutdown_connections 2024-03-12T03:17:33.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.561+0000 7f7efd428700 1 --2- 172.21.15.177:0/3678084993 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8103570 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:33.573 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.561+0000 7f7efd428700 1 -- 172.21.15.177:0/3678084993 >> 172.21.15.177:0/3678084993 conn(0x7f7ef80fab70 msgr2=0x7f7ef80fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.561+0000 7f7efd428700 1 -- 172.21.15.177:0/3678084993 shutdown_connections 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.561+0000 7f7efd428700 1 -- 172.21.15.177:0/3678084993 wait complete. 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.562+0000 7f7efd428700 1 Processor -- start 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.562+0000 7f7efd428700 1 -- start start 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.562+0000 7f7efd428700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8195b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.562+0000 7f7efd428700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef8196070 con 0x7f7ef8101180 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.562+0000 7f7ef6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8195b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.562+0000 7f7ef6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8195b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:53694/0 (socket says 172.21.15.177:53694) 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.562+0000 7f7ef6ffd700 1 -- 172.21.15.177:0/2376827123 learned_addr learned my addr 172.21.15.177:0/2376827123 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.563+0000 7f7ef6ffd700 1 -- 172.21.15.177:0/2376827123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee80095f0 con 0x7f7ef8101180 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.563+0000 7f7ef6ffd700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8195b30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009ed0 tx=0x7f7ee8004690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.563+0000 7f7ef4ff9700 1 -- 172.21.15.177:0/2376827123 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ee801d070 con 0x7f7ef8101180 2024-03-12T03:17:33.574 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.563+0000 7f7ef4ff9700 1 -- 172.21.15.177:0/2376827123 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ee800be20 con 0x7f7ef8101180 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.563+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef8196270 con 0x7f7ef8101180 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.563+0000 7f7ef4ff9700 1 -- 172.21.15.177:0/2376827123 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ee800f460 con 0x7f7ef8101180 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.563+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef8196710 con 0x7f7ef8101180 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.564+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef818f520 con 0x7f7ef8101180 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.564+0000 7f7ef4ff9700 1 -- 172.21.15.177:0/2376827123 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f7ee8022800 con 0x7f7ef8101180 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.564+0000 7f7ef4ff9700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7ee40385a0 0x7f7ee403aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.564+0000 7f7ef4ff9700 1 -- 172.21.15.177:0/2376827123 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7ee804c920 con 0x7f7ef8101180 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.564+0000 7f7eee7fc700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7ee40385a0 0x7f7ee403aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.565+0000 7f7eee7fc700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7ee40385a0 0x7f7ee403aa60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006fd0 tx=0x7f7ee0006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:33.575 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.569+0000 7f7ef4ff9700 1 -- 172.21.15.177:0/2376827123 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7ee802bba0 con 0x7f7ef8101180 2024-03-12T03:17:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.836+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ef80623e0 con 0x7f7ef8101180 2024-03-12T03:17:33.839 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.837+0000 7f7ef4ff9700 1 -- 172.21.15.177:0/2376827123 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f7ee802bba0 con 0x7f7ef8101180 2024-03-12T03:17:33.839 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:33.839 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7ee40385a0 msgr2=0x7f7ee403aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:33.841 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7ee40385a0 0x7f7ee403aa60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0006fd0 tx=0x7f7ee0006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 msgr2=0x7f7ef8195b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8195b30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009ed0 tx=0x7f7ee8004690 comp rx=0 tx=0).stop 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 shutdown_connections 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f7ee40385a0 0x7f7ee403aa60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 --2- 172.21.15.177:0/2376827123 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7ef8101180 0x7f7ef8195b30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 >> 172.21.15.177:0/2376827123 conn(0x7f7ef80fab70 msgr2=0x7f7ef80fb600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 shutdown_connections 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:33.839+0000 7f7efd428700 1 -- 172.21.15.177:0/2376827123 wait complete. 2024-03-12T03:17:33.842 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:35.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:34 smithi118 ceph-mon[29160]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:35.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:34 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/2376827123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:35.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:35.457 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:35.661 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:35.707 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.197+0000 7fa5854ee700 1 -- 172.21.15.177:0/2040242233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 msgr2=0x7fa580102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.197+0000 7fa5854ee700 1 --2- 172.21.15.177:0/2040242233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580102120 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa57800b0a0 tx=0x7fa57800b3b0 comp rx=0 tx=0).stop 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.198+0000 7fa5854ee700 1 -- 172.21.15.177:0/2040242233 shutdown_connections 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.198+0000 7fa5854ee700 1 --2- 172.21.15.177:0/2040242233 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580102120 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.198+0000 7fa5854ee700 1 -- 172.21.15.177:0/2040242233 >> 172.21.15.177:0/2040242233 conn(0x7fa5800fa320 msgr2=0x7fa5800fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.198+0000 7fa5854ee700 1 -- 172.21.15.177:0/2040242233 shutdown_connections 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.198+0000 7fa5854ee700 1 -- 172.21.15.177:0/2040242233 wait complete. 2024-03-12T03:17:37.201 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.198+0000 7fa5854ee700 1 Processor -- start 2024-03-12T03:17:37.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.198+0000 7fa5854ee700 1 -- start start 2024-03-12T03:17:37.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa5854ee700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580195ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:37.202 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa5854ee700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5801960e0 con 0x7fa580101d00 2024-03-12T03:17:37.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa57effd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580195ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:37.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa57effd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580195ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:53714/0 (socket says 172.21.15.177:53714) 2024-03-12T03:17:37.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa57effd700 1 -- 172.21.15.177:0/2814638980 learned_addr learned my addr 172.21.15.177:0/2814638980 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:37.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa57effd700 1 -- 172.21.15.177:0/2814638980 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578009d00 con 0x7fa580101d00 2024-03-12T03:17:37.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa57effd700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580195ba0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa5780062a0 tx=0x7fa57800bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:37.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.199+0000 7fa567fff700 1 -- 172.21.15.177:0/2814638980 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa57800e040 con 0x7fa580101d00 2024-03-12T03:17:37.203 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.200+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5801962e0 con 0x7fa580101d00 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.200+0000 7fa567fff700 1 -- 172.21.15.177:0/2814638980 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa578003d70 con 0x7fa580101d00 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.200+0000 7fa567fff700 1 -- 172.21.15.177:0/2814638980 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa57801c890 con 0x7fa580101d00 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.200+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa580196700 con 0x7fa580101d00 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.200+0000 7fa567fff700 1 -- 172.21.15.177:0/2814638980 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7fa57801cb50 con 0x7fa580101d00 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.200+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58018f550 con 0x7fa580101d00 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.201+0000 7fa567fff700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa56803c9a0 0x7fa56803ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.201+0000 7fa567fff700 1 -- 172.21.15.177:0/2814638980 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa57804c720 con 0x7fa580101d00 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.201+0000 7fa57e7fc700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa56803c9a0 0x7fa56803ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:37.204 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.201+0000 7fa57e7fc700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa56803c9a0 0x7fa56803ee60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa574006fd0 tx=0x7fa574006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:37.208 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.206+0000 7fa567fff700 1 -- 172.21.15.177:0/2814638980 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa57801a590 con 0x7fa580101d00 2024-03-12T03:17:37.211 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:36 smithi118 ceph-mon[29160]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:37.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.480+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5800619c0 con 0x7fa580101d00 2024-03-12T03:17:37.483 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.481+0000 7fa567fff700 1 -- 172.21.15.177:0/2814638980 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa57801a3b0 con 0x7fa580101d00 2024-03-12T03:17:37.484 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:37.484 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa56803c9a0 msgr2=0x7fa56803ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa56803c9a0 0x7fa56803ee60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa574006fd0 tx=0x7fa574006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 msgr2=0x7fa580195ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580195ba0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa5780062a0 tx=0x7fa57800bd40 comp rx=0 tx=0).stop 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 shutdown_connections 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa56803c9a0 0x7fa56803ee60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 --2- 172.21.15.177:0/2814638980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa580101d00 0x7fa580195ba0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 >> 172.21.15.177:0/2814638980 conn(0x7fa5800fa320 msgr2=0x7fa5800fae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:37.486 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 shutdown_connections 2024-03-12T03:17:37.487 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:37.483+0000 7fa5854ee700 1 -- 172.21.15.177:0/2814638980 wait complete. 2024-03-12T03:17:37.487 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:38.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:37 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/2814638980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:39.067 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:39.067 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:39.267 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:39.302 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:38 smithi118 ceph-mon[29160]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:39.312 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:40.645 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.642+0000 7fa54d821700 1 -- 172.21.15.177:0/3655130860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 msgr2=0x7fa5480ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:40.645 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.642+0000 7fa54d821700 1 --2- 172.21.15.177:0/3655130860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa5480ffa10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa53c009910 tx=0x7fa53c009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:40.645 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.642+0000 7fa54d821700 1 -- 172.21.15.177:0/3655130860 shutdown_connections 2024-03-12T03:17:40.645 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.642+0000 7fa54d821700 1 --2- 172.21.15.177:0/3655130860 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa5480ffa10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:40.646 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.642+0000 7fa54d821700 1 -- 172.21.15.177:0/3655130860 >> 172.21.15.177:0/3655130860 conn(0x7fa5480fab70 msgr2=0x7fa5480fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:40.646 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.642+0000 7fa54d821700 1 -- 172.21.15.177:0/3655130860 shutdown_connections 2024-03-12T03:17:40.646 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.642+0000 7fa54d821700 1 -- 172.21.15.177:0/3655130860 wait complete. 2024-03-12T03:17:40.646 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.643+0000 7fa54d821700 1 Processor -- start 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.643+0000 7fa54d821700 1 -- start start 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.643+0000 7fa54d821700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa548197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.643+0000 7fa54d821700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa548198330 con 0x7fa5480ff5f0 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.643+0000 7fa546ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa548197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.643+0000 7fa546ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa548197df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:53730/0 (socket says 172.21.15.177:53730) 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa546ffd700 1 -- 172.21.15.177:0/194274931 learned_addr learned my addr 172.21.15.177:0/194274931 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa546ffd700 1 -- 172.21.15.177:0/194274931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c0095f0 con 0x7fa5480ff5f0 2024-03-12T03:17:40.648 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa546ffd700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa548197df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa53c006010 tx=0x7fa53c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa54c81f700 1 -- 172.21.15.177:0/194274931 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa53c01d070 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa54c81f700 1 -- 172.21.15.177:0/194274931 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa53c005dc0 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa548198530 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa54c81f700 1 -- 172.21.15.177:0/194274931 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa53c022e90 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.644+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5481989d0 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.645+0000 7fa54c81f700 1 -- 172.21.15.177:0/194274931 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 45446+0+0 (secure 0 0 0) 0x7fa53c00f550 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.645+0000 7fa54c81f700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa530038530 0x7fa53003a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.645+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa548191f40 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.645+0000 7fa54c81f700 1 -- 172.21.15.177:0/194274931 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa53c04c710 con 0x7fa5480ff5f0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.645+0000 7fa5467fc700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa530038530 0x7fa53003a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:40.649 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.646+0000 7fa5467fc700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa530038530 0x7fa53003a9f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa538006fd0 tx=0x7fa538006e40 comp rx=0 tx=0).ready entity=mgr.14164 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:40.653 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.651+0000 7fa54c81f700 1 -- 172.21.15.177:0/194274931 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fa53c027070 con 0x7fa5480ff5f0 2024-03-12T03:17:40.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.919+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5480623e0 con 0x7fa5480ff5f0 2024-03-12T03:17:40.922 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.920+0000 7fa54c81f700 1 -- 172.21.15.177:0/194274931 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fa53c02a720 con 0x7fa5480ff5f0 2024-03-12T03:17:40.922 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:40.922 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:40.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.921+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa530038530 msgr2=0x7fa53003a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:40.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.921+0000 7fa54d821700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa530038530 0x7fa53003a9f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa538006fd0 tx=0x7fa538006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:40.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 msgr2=0x7fa548197df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:40.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa548197df0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa53c006010 tx=0x7fa53c009f80 comp rx=0 tx=0).stop 2024-03-12T03:17:40.924 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 shutdown_connections 2024-03-12T03:17:40.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7fa530038530 0x7fa53003a9f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:40.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 --2- 172.21.15.177:0/194274931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5480ff5f0 0x7fa548197df0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:40.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 >> 172.21.15.177:0/194274931 conn(0x7fa5480fab70 msgr2=0x7fa54818f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:40.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 shutdown_connections 2024-03-12T03:17:40.925 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:40.922+0000 7fa54d821700 1 -- 172.21.15.177:0/194274931 wait complete. 2024-03-12T03:17:40.925 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:41.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:40 smithi118 ceph-mon[29160]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:17:41.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:40 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:41.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:40 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:41.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:40 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:41.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:40 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:17:42.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:41 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/194274931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:42.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:41 smithi118 ceph-mon[29160]: from='mgr.14164 172.21.15.118:0/2242230209' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:17:42.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:41 smithi118 ceph-mon[29160]: mgrmap e14: smithi118.zzzuvl(active, since 70s) 2024-03-12T03:17:42.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:42.530 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:42.741 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:42.787 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.321+0000 7f269b346700 1 -- 172.21.15.177:0/1289699605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 msgr2=0x7f26941035d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.321+0000 7f269b346700 1 --2- 172.21.15.177:0/1289699605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f26941035d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f268c00b0a0 tx=0x7f268c00b3b0 comp rx=0 tx=0).stop 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.321+0000 7f269b346700 1 -- 172.21.15.177:0/1289699605 shutdown_connections 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.321+0000 7f269b346700 1 --2- 172.21.15.177:0/1289699605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f26941035d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.321+0000 7f269b346700 1 -- 172.21.15.177:0/1289699605 >> 172.21.15.177:0/1289699605 conn(0x7f26940fab70 msgr2=0x7f26940fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.321+0000 7f269b346700 1 -- 172.21.15.177:0/1289699605 shutdown_connections 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f269b346700 1 -- 172.21.15.177:0/1289699605 wait complete. 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f269b346700 1 Processor -- start 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f269b346700 1 -- start start 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f269b346700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f2694195b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:44.326 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f269b346700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694196040 con 0x7f26941011e0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f26990e2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f2694195b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f26990e2700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f2694195b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:51444/0 (socket says 172.21.15.177:51444) 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.322+0000 7f26990e2700 1 -- 172.21.15.177:0/2227370370 learned_addr learned my addr 172.21.15.177:0/2227370370 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.323+0000 7f26990e2700 1 -- 172.21.15.177:0/2227370370 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c009d00 con 0x7f26941011e0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.323+0000 7f26990e2700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f2694195b00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f268c015040 tx=0x7f268c00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.323+0000 7f268a7fc700 1 -- 172.21.15.177:0/2227370370 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268c00e040 con 0x7f26941011e0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.323+0000 7f268a7fc700 1 -- 172.21.15.177:0/2227370370 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f268c003d70 con 0x7f26941011e0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.323+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2694196240 con 0x7f26941011e0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.323+0000 7f268a7fc700 1 -- 172.21.15.177:0/2227370370 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f268c01c9e0 con 0x7f26941011e0 2024-03-12T03:17:44.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.323+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26941966e0 con 0x7f26941011e0 2024-03-12T03:17:44.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.324+0000 7f268a7fc700 1 -- 172.21.15.177:0/2227370370 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 14) v1 ==== 45460+0+0 (secure 0 0 0) 0x7f268c01cc50 con 0x7f26941011e0 2024-03-12T03:17:44.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.324+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f269418f520 con 0x7f26941011e0 2024-03-12T03:17:44.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.324+0000 7f268a7fc700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 0x7f268003ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:44.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.324+0000 7f268a7fc700 1 -- 172.21.15.177:0/2227370370 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f268c04c640 con 0x7f26941011e0 2024-03-12T03:17:44.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.324+0000 7f26988e1700 1 -- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 msgr2=0x7f268003ab40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/180481365 2024-03-12T03:17:44.329 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.324+0000 7f26988e1700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 0x7f268003ab40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:17:44.331 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.329+0000 7f268a7fc700 1 -- 172.21.15.177:0/2227370370 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f268c01a590 con 0x7f26941011e0 2024-03-12T03:17:44.527 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.525+0000 7f26988e1700 1 -- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 msgr2=0x7f268003ab40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/180481365 2024-03-12T03:17:44.528 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.525+0000 7f26988e1700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 0x7f268003ab40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T03:17:44.601 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.599+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26940623e0 con 0x7f26941011e0 2024-03-12T03:17:44.602 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.599+0000 7f268a7fc700 1 -- 172.21.15.177:0/2227370370 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f268c01a3b0 con 0x7f26941011e0 2024-03-12T03:17:44.602 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:44.602 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:44.604 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 msgr2=0x7f268003ab40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:17:44.604 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 0x7f268003ab40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:44.604 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 msgr2=0x7f2694195b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:44.604 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f2694195b00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f268c015040 tx=0x7f268c00bd40 comp rx=0 tx=0).stop 2024-03-12T03:17:44.604 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 shutdown_connections 2024-03-12T03:17:44.604 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f2680038680 0x7f268003ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:44.605 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 --2- 172.21.15.177:0/2227370370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26941011e0 0x7f2694195b00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:44.605 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.601+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 >> 172.21.15.177:0/2227370370 conn(0x7f26940fab70 msgr2=0x7f26940fb580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:44.605 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.602+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 shutdown_connections 2024-03-12T03:17:44.605 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:44.602+0000 7f269b346700 1 -- 172.21.15.177:0/2227370370 wait complete. 2024-03-12T03:17:44.605 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:45.132 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:44 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/2227370370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:46.169 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:46.169 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:46.375 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:46.421 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:47.974 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.970+0000 7f36ceba8700 1 -- 172.21.15.177:0/2054814883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 msgr2=0x7f36c8100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.970+0000 7f36ceba8700 1 --2- 172.21.15.177:0/2054814883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8100b10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f36c000b0a0 tx=0x7f36c000b3b0 comp rx=0 tx=0).stop 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.970+0000 7f36ceba8700 1 -- 172.21.15.177:0/2054814883 shutdown_connections 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.970+0000 7f36ceba8700 1 --2- 172.21.15.177:0/2054814883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8100b10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.970+0000 7f36ceba8700 1 -- 172.21.15.177:0/2054814883 >> 172.21.15.177:0/2054814883 conn(0x7f36c80fbcb0 msgr2=0x7f36c80fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.970+0000 7f36ceba8700 1 -- 172.21.15.177:0/2054814883 shutdown_connections 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.970+0000 7f36ceba8700 1 -- 172.21.15.177:0/2054814883 wait complete. 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36ceba8700 1 Processor -- start 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36ceba8700 1 -- start start 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36ceba8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8197d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36ceba8700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c81982d0 con 0x7f36c81006f0 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36cc944700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8197d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36cc944700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8197d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:51470/0 (socket says 172.21.15.177:51470) 2024-03-12T03:17:47.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36cc944700 1 -- 172.21.15.177:0/925838722 learned_addr learned my addr 172.21.15.177:0/925838722 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:47.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.971+0000 7f36cc944700 1 -- 172.21.15.177:0/925838722 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0009d00 con 0x7f36c81006f0 2024-03-12T03:17:47.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.972+0000 7f36cc944700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8197d90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36c00062a0 tx=0x7f36c000bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:47.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.972+0000 7f36c5ffb700 1 -- 172.21.15.177:0/925838722 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c000e040 con 0x7f36c81006f0 2024-03-12T03:17:47.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.972+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c81984d0 con 0x7f36c81006f0 2024-03-12T03:17:47.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.972+0000 7f36c5ffb700 1 -- 172.21.15.177:0/925838722 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f36c0003d70 con 0x7f36c81006f0 2024-03-12T03:17:47.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.972+0000 7f36c5ffb700 1 -- 172.21.15.177:0/925838722 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f36c001c890 con 0x7f36c81006f0 2024-03-12T03:17:47.976 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.972+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c8198970 con 0x7f36c81006f0 2024-03-12T03:17:47.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.973+0000 7f36c5ffb700 1 -- 172.21.15.177:0/925838722 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 14) v1 ==== 45460+0+0 (secure 0 0 0) 0x7f36c001cb50 con 0x7f36c81006f0 2024-03-12T03:17:47.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.973+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c8191920 con 0x7f36c81006f0 2024-03-12T03:17:47.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.973+0000 7f36c5ffb700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 0x7f36b403aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:47.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.973+0000 7f36c5ffb700 1 -- 172.21.15.177:0/925838722 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f36c004c760 con 0x7f36c81006f0 2024-03-12T03:17:47.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.973+0000 7f36c7fff700 1 -- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 msgr2=0x7f36b403aaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/180481365 2024-03-12T03:17:47.977 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.973+0000 7f36c7fff700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 0x7f36b403aaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:17:47.980 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:47.978+0000 7f36c5ffb700 1 -- 172.21.15.177:0/925838722 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f36c0024b50 con 0x7f36c81006f0 2024-03-12T03:17:48.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.174+0000 7f36c7fff700 1 -- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 msgr2=0x7f36b403aaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/180481365 2024-03-12T03:17:48.176 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.174+0000 7f36c7fff700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 0x7f36b403aaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T03:17:48.253 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.251+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f36c80623e0 con 0x7f36c81006f0 2024-03-12T03:17:48.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.251+0000 7f36c5ffb700 1 -- 172.21.15.177:0/925838722 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f36c80623e0 con 0x7f36c81006f0 2024-03-12T03:17:48.254 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:48.254 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:48.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.253+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 msgr2=0x7f36b403aaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:17:48.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.253+0000 7f36ceba8700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 0x7f36b403aaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:48.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.253+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 msgr2=0x7f36c8197d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:48.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.253+0000 7f36ceba8700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8197d90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f36c00062a0 tx=0x7f36c000bd40 comp rx=0 tx=0).stop 2024-03-12T03:17:48.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.254+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 shutdown_connections 2024-03-12T03:17:48.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.254+0000 7f36ceba8700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:6800/180481365,v1:172.21.15.118:6801/180481365] conn(0x7f36b40385e0 0x7f36b403aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:48.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.254+0000 7f36ceba8700 1 --2- 172.21.15.177:0/925838722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f36c81006f0 0x7f36c8197d90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:48.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.254+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 >> 172.21.15.177:0/925838722 conn(0x7f36c80fbcb0 msgr2=0x7f36c80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:48.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.254+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 shutdown_connections 2024-03-12T03:17:48.257 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:48.254+0000 7f36ceba8700 1 -- 172.21.15.177:0/925838722 wait complete. 2024-03-12T03:17:48.257 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:48.503 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:48 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/925838722' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: Active manager daemon smithi118.zzzuvl restarted 2024-03-12T03:17:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: Activating manager daemon smithi118.zzzuvl 2024-03-12T03:17:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: osdmap e5: 0 total, 0 up, 0 in 2024-03-12T03:17:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: mgrmap e15: smithi118.zzzuvl(active, starting, since 0.00331207s) 2024-03-12T03:17:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:17:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr metadata", "who": "smithi118.zzzuvl", "id": "smithi118.zzzuvl"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: Manager daemon smithi118.zzzuvl is now available 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi118.zzzuvl/trash_purge_schedule"}]: dispatch 2024-03-12T03:17:49.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:49.829 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:49.829 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:50.024 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:50.071 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:50.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:50 smithi118 ceph-mon[29160]: [12/Mar/2024:03:17:49] ENGINE Bus STARTING 2024-03-12T03:17:50.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:50 smithi118 ceph-mon[29160]: [12/Mar/2024:03:17:49] ENGINE Serving on https://172.21.15.118:7150 2024-03-12T03:17:50.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:50 smithi118 ceph-mon[29160]: [12/Mar/2024:03:17:49] ENGINE Serving on http://172.21.15.118:8765 2024-03-12T03:17:50.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:50 smithi118 ceph-mon[29160]: [12/Mar/2024:03:17:49] ENGINE Bus STARTED 2024-03-12T03:17:50.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:50 smithi118 ceph-mon[29160]: mgrmap e16: smithi118.zzzuvl(active, since 1.00528s) 2024-03-12T03:17:50.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:50 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:50.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:50 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.421+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1018137366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 msgr2=0x7f92ac1020f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.421+0000 7f92b0d8a700 1 --2- 172.21.15.177:0/1018137366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac1020f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f929c009910 tx=0x7f929c009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.421+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1018137366 shutdown_connections 2024-03-12T03:17:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.421+0000 7f92b0d8a700 1 --2- 172.21.15.177:0/1018137366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac1020f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.421+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1018137366 >> 172.21.15.177:0/1018137366 conn(0x7f92ac0fa2d0 msgr2=0x7f92ac0fc750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.421+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1018137366 shutdown_connections 2024-03-12T03:17:51.423 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.421+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1018137366 wait complete. 2024-03-12T03:17:51.424 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.422+0000 7f92b0d8a700 1 Processor -- start 2024-03-12T03:17:51.424 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.422+0000 7f92b0d8a700 1 -- start start 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.422+0000 7f92b0d8a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.422+0000 7f92b0d8a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac1960d0 con 0x7f92ac101cd0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.422+0000 7f92aa59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.422+0000 7f92aa59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:51490/0 (socket says 172.21.15.177:51490) 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.422+0000 7f92aa59c700 1 -- 172.21.15.177:0/1925773059 learned_addr learned my addr 172.21.15.177:0/1925773059 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.423+0000 7f92aa59c700 1 -- 172.21.15.177:0/1925773059 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c0095f0 con 0x7f92ac101cd0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.423+0000 7f92aa59c700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac195b90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f929c006010 tx=0x7f929c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.423+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f929c01d070 con 0x7f92ac101cd0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.423+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac1962d0 con 0x7f92ac101cd0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.423+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f929c005dc0 con 0x7f92ac101cd0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.423+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f929c022e90 con 0x7f92ac101cd0 2024-03-12T03:17:51.425 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.423+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac196770 con 0x7f92ac101cd0 2024-03-12T03:17:51.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.424+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 45310+0+0 (secure 0 0 0) 0x7f929c00f4b0 con 0x7f92ac101cd0 2024-03-12T03:17:51.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.424+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac18f520 con 0x7f92ac101cd0 2024-03-12T03:17:51.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.424+0000 7f92a3fff700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9298038410 0x7f929803a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:51.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.424+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f929c04c680 con 0x7f92ac101cd0 2024-03-12T03:17:51.426 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.424+0000 7f92a1d9b700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9298038410 0x7f929803a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:51.427 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.425+0000 7f92a1d9b700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9298038410 0x7f929803a8d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9294006fd0 tx=0x7f9294006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:51.431 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.429+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f929c017440 con 0x7f92ac101cd0 2024-03-12T03:17:51.521 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.520+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 17) v1 ==== 45504+0+0 (secure 0 0 0) 0x7f929c022750 con 0x7f92ac101cd0 2024-03-12T03:17:51.712 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.710+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92ac0623e0 con 0x7f92ac101cd0 2024-03-12T03:17:51.712 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.710+0000 7f92a3fff700 1 -- 172.21.15.177:0/1925773059 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f929c027020 con 0x7f92ac101cd0 2024-03-12T03:17:51.712 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:51.712 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:51.714 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.712+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9298038410 msgr2=0x7f929803a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:51.714 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.712+0000 7f92b0d8a700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9298038410 0x7f929803a8d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9294006fd0 tx=0x7f9294006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:51.714 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 msgr2=0x7f92ac195b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:51.714 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac195b90 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f929c006010 tx=0x7f929c009f80 comp rx=0 tx=0).stop 2024-03-12T03:17:51.714 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 shutdown_connections 2024-03-12T03:17:51.715 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9298038410 0x7f929803a8d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:51.715 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 --2- 172.21.15.177:0/1925773059 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92ac101cd0 0x7f92ac195b90 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:51.715 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 >> 172.21.15.177:0/1925773059 conn(0x7f92ac0fa2d0 msgr2=0x7f92ac0fae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:51.715 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 shutdown_connections 2024-03-12T03:17:51.715 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:51.713+0000 7f92b0d8a700 1 -- 172.21.15.177:0/1925773059 wait complete. 2024-03-12T03:17:51.715 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:52 smithi118 ceph-mon[29160]: mgrmap e17: smithi118.zzzuvl(active, since 3s) 2024-03-12T03:17:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:52 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1925773059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:53.905 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:53.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:17:54.616 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:54.663 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:17:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-12T03:17:56.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:55 smithi118 ceph-mon[29160]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-12T03:17:56.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.249+0000 7fe7c3809700 1 -- 172.21.15.177:0/2907675985 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 msgr2=0x7fe7bc105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:56.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.249+0000 7fe7c3809700 1 --2- 172.21.15.177:0/2907675985 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc105a40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8009910 tx=0x7fe7b8009c20 comp rx=0 tx=0).stop 2024-03-12T03:17:56.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.249+0000 7fe7c3809700 1 -- 172.21.15.177:0/2907675985 shutdown_connections 2024-03-12T03:17:56.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.249+0000 7fe7c3809700 1 --2- 172.21.15.177:0/2907675985 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc105a40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:56.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.249+0000 7fe7c3809700 1 -- 172.21.15.177:0/2907675985 >> 172.21.15.177:0/2907675985 conn(0x7fe7bc0fb0e0 msgr2=0x7fe7bc0fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:56.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.249+0000 7fe7c3809700 1 -- 172.21.15.177:0/2907675985 shutdown_connections 2024-03-12T03:17:56.252 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.249+0000 7fe7c3809700 1 -- 172.21.15.177:0/2907675985 wait complete. 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.250+0000 7fe7c3809700 1 Processor -- start 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.250+0000 7fe7c3809700 1 -- start start 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.250+0000 7fe7c3809700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc194d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.250+0000 7fe7c3809700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc195240 con 0x7fe7bc105660 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.250+0000 7fe7c15a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc194d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.250+0000 7fe7c15a5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc194d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:39352/0 (socket says 172.21.15.177:39352) 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.250+0000 7fe7c15a5700 1 -- 172.21.15.177:0/2519056437 learned_addr learned my addr 172.21.15.177:0/2519056437 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.251+0000 7fe7c15a5700 1 -- 172.21.15.177:0/2519056437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b80095f0 con 0x7fe7bc105660 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.251+0000 7fe7c15a5700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc194d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8005490 tx=0x7fe7b8005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:56.254 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.251+0000 7fe7b27fc700 1 -- 172.21.15.177:0/2519056437 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b801d070 con 0x7fe7bc105660 2024-03-12T03:17:56.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.251+0000 7fe7b27fc700 1 -- 172.21.15.177:0/2519056437 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7b8005700 con 0x7fe7bc105660 2024-03-12T03:17:56.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.251+0000 7fe7b27fc700 1 -- 172.21.15.177:0/2519056437 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b8003950 con 0x7fe7bc105660 2024-03-12T03:17:56.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.251+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc1954a0 con 0x7fe7bc105660 2024-03-12T03:17:56.255 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.251+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc1959a0 con 0x7fe7bc105660 2024-03-12T03:17:56.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.252+0000 7fe7b27fc700 1 -- 172.21.15.177:0/2519056437 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) v1 ==== 45504+0+0 (secure 0 0 0) 0x7fe7b8003c10 con 0x7fe7bc105660 2024-03-12T03:17:56.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.252+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc100cd0 con 0x7fe7bc105660 2024-03-12T03:17:56.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.252+0000 7fe7b27fc700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe7a8038690 0x7fe7a803ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:17:56.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.252+0000 7fe7b27fc700 1 -- 172.21.15.177:0/2519056437 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fe7b804c1d0 con 0x7fe7bc105660 2024-03-12T03:17:56.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.252+0000 7fe7c0da4700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe7a8038690 0x7fe7a803ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:17:56.256 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.253+0000 7fe7c0da4700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe7a8038690 0x7fe7a803ab50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:17:56.260 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.258+0000 7fe7b27fc700 1 -- 172.21.15.177:0/2519056437 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe7b8029540 con 0x7fe7bc105660 2024-03-12T03:17:56.560 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.557+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7bc0623e0 con 0x7fe7bc105660 2024-03-12T03:17:56.560 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.558+0000 7fe7b27fc700 1 -- 172.21.15.177:0/2519056437 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fe7bc0623e0 con 0x7fe7bc105660 2024-03-12T03:17:56.561 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:17:56.561 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe7a8038690 msgr2=0x7fe7a803ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe7a8038690 0x7fe7a803ab50 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac006fd0 tx=0x7fe7ac006e40 comp rx=0 tx=0).stop 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 msgr2=0x7fe7bc194d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc194d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe7b8005490 tx=0x7fe7b8005570 comp rx=0 tx=0).stop 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 shutdown_connections 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe7a8038690 0x7fe7a803ab50 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 --2- 172.21.15.177:0/2519056437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7bc105660 0x7fe7bc194d00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 >> 172.21.15.177:0/2519056437 conn(0x7fe7bc0fb0e0 msgr2=0x7fe7bc0fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 shutdown_connections 2024-03-12T03:17:56.563 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:17:56.560+0000 7fe7c3809700 1 -- 172.21.15.177:0/2519056437 wait complete. 2024-03-12T03:17:56.564 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:17:57.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: Updating smithi118:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:17:57.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: Updating smithi177:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:17:57.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:17:57.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:17:57.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: Updating smithi118:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: Updating smithi177:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi177", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi177", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: Deploying daemon ceph-exporter.smithi177 on smithi177 2024-03-12T03:17:57.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:56 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/2519056437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:17:58.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:17:58.842 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:17:59.049 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi177", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi177", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:00.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:17:59 smithi118 ceph-mon[29160]: Deploying daemon crash.smithi177 on smithi177 2024-03-12T03:18:01.756 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 -- 172.21.15.177:0/4218132701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 msgr2=0x7f6b140735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 --2- 172.21.15.177:0/4218132701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b140735c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6b04009910 tx=0x7f6b04009c20 comp rx=0 tx=0).stop 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 -- 172.21.15.177:0/4218132701 shutdown_connections 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 --2- 172.21.15.177:0/4218132701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b140735c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 -- 172.21.15.177:0/4218132701 >> 172.21.15.177:0/4218132701 conn(0x7f6b140fc180 msgr2=0x7f6b140fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 -- 172.21.15.177:0/4218132701 shutdown_connections 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 -- 172.21.15.177:0/4218132701 wait complete. 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 Processor -- start 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.734+0000 7f6b1a627700 1 -- start start 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.735+0000 7f6b1a627700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b141981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:01.757 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.735+0000 7f6b1a627700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14198700 con 0x7f6b140731e0 2024-03-12T03:18:01.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.735+0000 7f6b13fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b141981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:01.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.735+0000 7f6b13fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b141981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:39374/0 (socket says 172.21.15.177:39374) 2024-03-12T03:18:01.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.735+0000 7f6b13fff700 1 -- 172.21.15.177:0/3292622631 learned_addr learned my addr 172.21.15.177:0/3292622631 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:18:01.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.735+0000 7f6b13fff700 1 -- 172.21.15.177:0/3292622631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b040095f0 con 0x7f6b140731e0 2024-03-12T03:18:01.758 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.736+0000 7f6b13fff700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b141981c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6b04005490 tx=0x7f6b04005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.736+0000 7f6b117fa700 1 -- 172.21.15.177:0/3292622631 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b0401d070 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.736+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b14198900 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.736+0000 7f6b117fa700 1 -- 172.21.15.177:0/3292622631 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b04005700 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.736+0000 7f6b117fa700 1 -- 172.21.15.177:0/3292622631 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b04003950 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.736+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b14198d20 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.737+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14074890 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.737+0000 7f6b117fa700 1 -- 172.21.15.177:0/3292622631 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) v1 ==== 45504+0+0 (secure 0 0 0) 0x7f6b04004240 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.737+0000 7f6b117fa700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6afc038690 0x7f6afc03ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.737+0000 7f6b117fa700 1 -- 172.21.15.177:0/3292622631 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f6b0404c390 con 0x7f6b140731e0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.737+0000 7f6b137fe700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6afc038690 0x7f6afc03ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.738+0000 7f6b137fe700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6afc038690 0x7f6afc03ab50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b08006fd0 tx=0x7f6b08006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:01.759 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:01.742+0000 7f6b117fa700 1 -- 172.21.15.177:0/3292622631 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6b0402a590 con 0x7f6b140731e0 2024-03-12T03:18:02.044 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.041+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b1419cab0 con 0x7f6b140731e0 2024-03-12T03:18:02.045 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.042+0000 7f6b117fa700 1 -- 172.21.15.177:0/3292622631 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7f6b04026070 con 0x7f6b140731e0 2024-03-12T03:18:02.045 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:18:02.045 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:18:02.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6afc038690 msgr2=0x7f6afc03ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:02.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6afc038690 0x7f6afc03ab50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b08006fd0 tx=0x7f6b08006e40 comp rx=0 tx=0).stop 2024-03-12T03:18:02.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 msgr2=0x7f6b141981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:02.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b141981c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6b04005490 tx=0x7f6b04005570 comp rx=0 tx=0).stop 2024-03-12T03:18:02.047 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 shutdown_connections 2024-03-12T03:18:02.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6afc038690 0x7f6afc03ab50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:02.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 --2- 172.21.15.177:0/3292622631 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6b140731e0 0x7f6b141981c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:02.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 >> 172.21.15.177:0/3292622631 conn(0x7f6b140fc180 msgr2=0x7f6b14104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:02.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 shutdown_connections 2024-03-12T03:18:02.048 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:02.045+0000 7f6b1a627700 1 -- 172.21.15.177:0/3292622631 wait complete. 2024-03-12T03:18:02.048 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:18:02.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:02 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/3292622631' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:18:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:03 smithi118 ceph-mon[29160]: Deploying daemon node-exporter.smithi177 on smithi177 2024-03-12T03:18:03.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:18:03.763 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:18:03.964 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:18:04.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:04.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:06.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.895+0000 7fdbe32ae700 1 -- 172.21.15.177:0/3365455269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 msgr2=0x7fdbdc106a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:06.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.895+0000 7fdbe32ae700 1 --2- 172.21.15.177:0/3365455269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc106a50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009910 tx=0x7fdbd8009c20 comp rx=0 tx=0).stop 2024-03-12T03:18:06.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.895+0000 7fdbe32ae700 1 -- 172.21.15.177:0/3365455269 shutdown_connections 2024-03-12T03:18:06.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.895+0000 7fdbe32ae700 1 --2- 172.21.15.177:0/3365455269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc106a50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:06.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.895+0000 7fdbe32ae700 1 -- 172.21.15.177:0/3365455269 >> 172.21.15.177:0/3365455269 conn(0x7fdbdc075110 msgr2=0x7fdbdc075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:06.899 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.895+0000 7fdbe32ae700 1 -- 172.21.15.177:0/3365455269 shutdown_connections 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.895+0000 7fdbe32ae700 1 -- 172.21.15.177:0/3365455269 wait complete. 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.896+0000 7fdbe32ae700 1 Processor -- start 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.896+0000 7fdbe32ae700 1 -- start start 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.896+0000 7fdbe32ae700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc0ff720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.896+0000 7fdbe32ae700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc101590 con 0x7fdbdc106670 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.896+0000 7fdbe104a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc0ff720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.896+0000 7fdbe104a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc0ff720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.177:46404/0 (socket says 172.21.15.177:46404) 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.896+0000 7fdbe104a700 1 -- 172.21.15.177:0/942571846 learned_addr learned my addr 172.21.15.177:0/942571846 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:18:06.900 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.897+0000 7fdbe104a700 1 -- 172.21.15.177:0/942571846 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd80095f0 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.897+0000 7fdbe104a700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc0ff720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8005490 tx=0x7fdbd8005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.897+0000 7fdbd27fc700 1 -- 172.21.15.177:0/942571846 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbd801d070 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.897+0000 7fdbd27fc700 1 -- 172.21.15.177:0/942571846 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbd8005700 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.897+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc0ffc60 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.897+0000 7fdbd27fc700 1 -- 172.21.15.177:0/942571846 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbd80038e0 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.897+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc1000e0 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.898+0000 7fdbd27fc700 1 -- 172.21.15.177:0/942571846 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) v1 ==== 45504+0+0 (secure 0 0 0) 0x7fdbd8003ba0 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.898+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc108310 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.898+0000 7fdbd27fc700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdbc8038690 0x7fdbc803ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.898+0000 7fdbd27fc700 1 -- 172.21.15.177:0/942571846 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fdbd804c350 con 0x7fdbdc106670 2024-03-12T03:18:06.902 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.899+0000 7fdbe0849700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdbc8038690 0x7fdbc803ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:06.903 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.899+0000 7fdbe0849700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdbc8038690 0x7fdbc803ab50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc006fd0 tx=0x7fdbcc006e40 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:06.906 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:06.904+0000 7fdbd27fc700 1 -- 172.21.15.177:0/942571846 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdbd8029950 con 0x7fdbdc106670 2024-03-12T03:18:07.192 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.189+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbdc101db0 con 0x7fdbdc106670 2024-03-12T03:18:07.193 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.190+0000 7fdbd27fc700 1 -- 172.21.15.177:0/942571846 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+748 (secure 0 0 0) 0x7fdbd8026070 con 0x7fdbdc106670 2024-03-12T03:18:07.193 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:18:07.193 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:15:20.539712Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:18:07.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.192+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdbc8038690 msgr2=0x7fdbc803ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:07.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.192+0000 7fdbe32ae700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdbc8038690 0x7fdbc803ab50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc006fd0 tx=0x7fdbcc006e40 comp rx=0 tx=0).stop 2024-03-12T03:18:07.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.192+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 msgr2=0x7fdbdc0ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:07.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.192+0000 7fdbe32ae700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc0ff720 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8005490 tx=0x7fdbd8005570 comp rx=0 tx=0).stop 2024-03-12T03:18:07.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.192+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 shutdown_connections 2024-03-12T03:18:07.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.192+0000 7fdbe32ae700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdbc8038690 0x7fdbc803ab50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:07.195 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.193+0000 7fdbe32ae700 1 --2- 172.21.15.177:0/942571846 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbdc106670 0x7fdbdc0ff720 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:07.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.193+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 >> 172.21.15.177:0/942571846 conn(0x7fdbdc075110 msgr2=0x7fdbdc102170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:07.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.193+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 shutdown_connections 2024-03-12T03:18:07.196 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:07.193+0000 7fdbe32ae700 1 -- 172.21.15.177:0/942571846 wait complete. 2024-03-12T03:18:07.196 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-12T03:18:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi177.tmmsuz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:18:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi177.tmmsuz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:18:07.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:18:07.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:07.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: Deploying daemon mgr.smithi177.tmmsuz on smithi177 2024-03-12T03:18:07.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:07 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/942571846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:18:09.766 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-12T03:18:09.767 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:18:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:11.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:10 smithi118 ceph-mon[29160]: Deploying daemon mon.smithi177 on smithi177 2024-03-12T03:18:11.558 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:18:12.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:11 smithi118 ceph-mon[29160]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:13.563 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 podman[37113]: 2024-03-12 03:18:13.397350796 +0000 UTC m=+0.030679037 image pull fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6 2024-03-12T03:18:13.563 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 podman[37113]: 2024-03-12 03:18:13.560533738 +0000 UTC m=+0.193861973 container create 0d8acd4f44b3c5b734a626f5fc0440f830426c8831a22fe98afe04ff0c16d265 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi177, GIT_CLEAN=True, RELEASE=reef-62a11b3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T03:18:13.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:13 smithi118 ceph-mon[29160]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:14.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 podman[37113]: 2024-03-12 03:18:13.813309664 +0000 UTC m=+0.446637914 container init 0d8acd4f44b3c5b734a626f5fc0440f830426c8831a22fe98afe04ff0c16d265 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi177, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=reef-62a11b3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:18:14.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 podman[37113]: 2024-03-12 03:18:13.817030451 +0000 UTC m=+0.450358687 container start 0d8acd4f44b3c5b734a626f5fc0440f830426c8831a22fe98afe04ff0c16d265 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi177, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=reef-62a11b3, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T03:18:14.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:18:14.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable), process ceph-mon, pid 2 2024-03-12T03:18:14.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: pidfile_write: ignore empty --pid-file 2024-03-12T03:18:14.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: load: jerasure load: lrc 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Git sha 0 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Compile date 2024-03-11 17:58:16 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: DB SUMMARY 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: DB Session ID: M65FKWTUAOFHB5JADKKJ 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi177/store.db dir, Total Num: 0, files: 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi177/store.db: 000004.log size: 511 ; 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:18:14.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.env: 0x563c7ec15720 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.info_log: 0x563c8066b380 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.statistics: (nil) 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.use_fsync: 0 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:18:14.205 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.db_log_dir: 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.wal_dir: 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:18:14.206 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.write_buffer_manager: 0x563c7f8e4140 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.unordered_write: 0 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:18:14.207 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:18:14.208 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:18:14.208 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:18:14.208 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.row_cache: None 2024-03-12T03:18:14.208 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.wal_filter: None 2024-03-12T03:18:14.209 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:18:14.209 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:18:14.209 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:18:14.209 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:18:14.209 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.wal_compression: 0 2024-03-12T03:18:14.209 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:18:14.210 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_open_files: -1 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:18:14.211 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Compression algorithms supported: 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kZSTD supported: 0 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi177/store.db/MANIFEST-000005 2024-03-12T03:18:14.212 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.merge_operator: 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_filter: None 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563c8066b4a0) 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: index_type: 0 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: data_block_index_type: 0 2024-03-12T03:18:14.213 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: index_shortening: 1 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: checksum: 4 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: no_block_cache: 0 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_cache: 0x563c7f9851f0 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_cache_options: 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: capacity : 536870912 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: num_shard_bits : 4 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: strict_capacity_limit : 0 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_cache_compressed: (nil) 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: persistent_cache: (nil) 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_size: 4096 2024-03-12T03:18:14.214 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_size_deviation: 10 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_restart_interval: 16 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: index_block_restart_interval: 1 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: metadata_block_size: 4096 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: partition_filters: 0 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: use_delta_encoding: 1 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: filter_policy: bloomfilter 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: whole_key_filtering: 1 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: verify_compression: 0 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: format_version: 5 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: enable_index_compression: 1 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: block_align: 0 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: max_auto_readahead_size: 262144 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: prepopulate_block_cache: 0 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:18:14.215 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression: NoCompression 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.num_levels: 7 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:18:14.216 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:18:14.217 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:18:14.218 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:18:14.219 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:18:14.220 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.ttl: 2592000 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi177/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:18:14.221 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: aab93355-9d56-4bb5-9723-9c83b9b3583a 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213493846979, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213493847504, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710213493, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "aab93355-9d56-4bb5-9723-9c83b9b3583a", "db_session_id": "M65FKWTUAOFHB5JADKKJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710213493847602, "job": 1, "event": "recovery_finished"} 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi177/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563c7fa22000 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: DB pointer 0x563c7fa10000 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: ** DB Stats ** 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:18:14.222 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:18:14.223 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Block cache BinnedLRUCache@0x563c7f9851f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: 2024-03-12T03:18:14.224 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177 does not exist in monmap, will attempt to join an existing cluster 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: using public_addr v2:172.21.15.177:0/0 -> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: starting mon.smithi177 rank -1 at public addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] at bind addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi177 fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(???) e0 preinit fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).mds e1 new map 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).mds e1 print_map 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: e1 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: legacy client fscid: -1 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout: No filesystems configured 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:18:14.225 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 ceph-mon[37140]: mon.smithi177@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 bash[37113]: 0d8acd4f44b3c5b734a626f5fc0440f830426c8831a22fe98afe04ff0c16d265 2024-03-12T03:18:14.226 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:13 smithi177 systemd[1]: Started Ceph mon.smithi177 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T03:18:19.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: mon.smithi118 calling monitor election 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: mon.smithi177 calling monitor election 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/crt"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: mon.smithi118 is new leader, mons smithi118,smithi177 in quorum (ranks 0,1) 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: monmap e2: 2 mons at {smithi118=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],smithi177=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: fsmap 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: osdmap e5: 0 total, 0 up, 0 in 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: mgrmap e17: smithi118.zzzuvl(active, since 30s) 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: Standby manager daemon smithi177.tmmsuz started 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: overall HEALTH_OK 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/key"}]: dispatch 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:19.204 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:18:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: mon.smithi118 calling monitor election 2024-03-12T03:18:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: mon.smithi177 calling monitor election 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/crt"}]: dispatch 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: mon.smithi118 is new leader, mons smithi118,smithi177 in quorum (ranks 0,1) 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: monmap e2: 2 mons at {smithi118=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],smithi177=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: fsmap 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: osdmap e5: 0 total, 0 up, 0 in 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: mgrmap e17: smithi118.zzzuvl(active, since 30s) 2024-03-12T03:18:19.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: Standby manager daemon smithi177.tmmsuz started 2024-03-12T03:18:19.313 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:18:19.313 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: overall HEALTH_OK 2024-03-12T03:18:19.313 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/key"}]: dispatch 2024-03-12T03:18:19.313 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:19.313 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/71279523' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:18:19.313 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:19.313 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:20.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:20.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:19 smithi177 ceph-mon[37140]: mgrmap e18: smithi118.zzzuvl(active, since 30s), standbys: smithi177.tmmsuz 2024-03-12T03:18:20.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr metadata", "who": "smithi177.tmmsuz", "id": "smithi177.tmmsuz"}]: dispatch 2024-03-12T03:18:20.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:20.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:20.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:20.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:20.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:19 smithi118 ceph-mon[29160]: mgrmap e18: smithi118.zzzuvl(active, since 30s), standbys: smithi177.tmmsuz 2024-03-12T03:18:20.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr metadata", "who": "smithi177.tmmsuz", "id": "smithi177.tmmsuz"}]: dispatch 2024-03-12T03:18:20.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:20.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:20.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.339+0000 7f7722bdf700 1 -- 172.21.15.177:0/2294726147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 msgr2=0x7f770c008710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.339+0000 7f7722bdf700 1 --2- 172.21.15.177:0/2294726147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 0x7f770c008710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f771c0766b0 tx=0x7f771000fe90 comp rx=0 tx=0).stop 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.339+0000 7f7722bdf700 1 -- 172.21.15.177:0/2294726147 shutdown_connections 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.339+0000 7f7722bdf700 1 --2- 172.21.15.177:0/2294726147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 0x7f770c008710 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.339+0000 7f7722bdf700 1 -- 172.21.15.177:0/2294726147 >> 172.21.15.177:0/2294726147 conn(0x7f771c075110 msgr2=0x7f771c075520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.339+0000 7f7722bdf700 1 -- 172.21.15.177:0/2294726147 shutdown_connections 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.339+0000 7f7722bdf700 1 -- 172.21.15.177:0/2294726147 wait complete. 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f7722bdf700 1 Processor -- start 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f7722bdf700 1 -- start start 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f7722bdf700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 0x7f771c0ff720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f7722bdf700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 0x7f771c1000e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:20.344 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f7722bdf700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c101670 con 0x7f770c0062a0 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f7722bdf700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c1a22e0 con 0x7f771c0ffc60 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f771bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 0x7f771c1000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f771bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 0x7f771c1000e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:34586/0 (socket says 172.21.15.177:34586) 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f771bfff700 1 -- 172.21.15.177:0/256989908 learned_addr learned my addr 172.21.15.177:0/256989908 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f771bfff700 1 -- 172.21.15.177:0/256989908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 msgr2=0x7f771c1000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f771bfff700 1 -- 172.21.15.177:0/256989908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 msgr2=0x7f771c1000e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f771bfff700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 0x7f771c1000e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f771bfff700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 0x7f771c1000e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f772097b700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 0x7f771c0ff720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f772097b700 1 -- 172.21.15.177:0/256989908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 msgr2=0x7f771c1000e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f772097b700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 0x7f771c1000e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:20.345 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.340+0000 7f772097b700 1 -- 172.21.15.177:0/256989908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771000f8e0 con 0x7f770c0062a0 2024-03-12T03:18:20.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.341+0000 7f772097b700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 0x7f771c0ff720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f771c0766b0 tx=0x7f7710005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:20.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.341+0000 7f7719ffb700 1 -- 172.21.15.177:0/256989908 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7710005da0 con 0x7f770c0062a0 2024-03-12T03:18:20.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.341+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c1a2480 con 0x7f770c0062a0 2024-03-12T03:18:20.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.341+0000 7f7719ffb700 1 -- 172.21.15.177:0/256989908 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7710012d30 con 0x7f770c0062a0 2024-03-12T03:18:20.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.341+0000 7f7719ffb700 1 -- 172.21.15.177:0/256989908 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7710003e50 con 0x7f770c0062a0 2024-03-12T03:18:20.346 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.341+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1a2940 con 0x7f770c0062a0 2024-03-12T03:18:20.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.342+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c101db0 con 0x7f770c0062a0 2024-03-12T03:18:20.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.343+0000 7f7719ffb700 1 -- 172.21.15.177:0/256989908 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) v1 ==== 90470+0+0 (secure 0 0 0) 0x7f771001d480 con 0x7f770c0062a0 2024-03-12T03:18:20.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.343+0000 7f7719ffb700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f770006c8b0 0x7f770006ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:20.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.343+0000 7f7719ffb700 1 -- 172.21.15.177:0/256989908 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f771008e0a0 con 0x7f770c0062a0 2024-03-12T03:18:20.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.343+0000 7f771bfff700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f770006c8b0 0x7f770006ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:20.347 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.344+0000 7f771bfff700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f770006c8b0 0x7f770006ed70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7708009180 tx=0x7f7708008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:20.350 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.348+0000 7f7719ffb700 1 -- 172.21.15.177:0/256989908 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7710059790 con 0x7f770c0062a0 2024-03-12T03:18:20.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.631+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f771c1a2b60 con 0x7f770c0062a0 2024-03-12T03:18:20.634 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.632+0000 7f7719ffb700 1 -- 172.21.15.177:0/256989908 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1026 (secure 0 0 0) 0x7f7710035090 con 0x7f770c0062a0 2024-03-12T03:18:20.635 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-12T03:18:20.635 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","modified":"2024-03-12T03:18:13.875584Z","created":"2024-03-12T03:15:20.539712Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi118","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi177","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T03:18:20.636 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f770006c8b0 msgr2=0x7f770006ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f770006c8b0 0x7f770006ed70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7708009180 tx=0x7f7708008040 comp rx=0 tx=0).stop 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 msgr2=0x7f771c0ff720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 0x7f771c0ff720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f771c0766b0 tx=0x7f7710005690 comp rx=0 tx=0).stop 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 shutdown_connections 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f770006c8b0 0x7f770006ed70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f770c0062a0 0x7f771c0ff720 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 --2- 172.21.15.177:0/256989908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f771c0ffc60 0x7f771c1000e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 >> 172.21.15.177:0/256989908 conn(0x7f771c075110 msgr2=0x7f771c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 shutdown_connections 2024-03-12T03:18:20.637 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:18:20.634+0000 7f7722bdf700 1 -- 172.21.15.177:0/256989908 wait complete. 2024-03-12T03:18:20.638 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-03-12T03:18:21.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:20 smithi118 ceph-mon[29160]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:21.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:20 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/256989908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:18:21.335 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:20 smithi177 ceph-mon[37140]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:21.335 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:20 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/256989908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:18:21.356 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T03:18:21.356 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T03:18:21.581 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.673 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: Updating smithi118:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: Updating smithi177:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:18:22.674 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:22.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: Updating smithi118:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: Updating smithi177:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:18:22.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:18:22.704 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:23.621 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:23 smithi118 ceph-mon[29160]: Reconfiguring mon.smithi118 (unknown last config time)... 2024-03-12T03:18:23.622 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:23 smithi118 ceph-mon[29160]: Reconfiguring daemon mon.smithi118 on smithi118 2024-03-12T03:18:23.622 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:23 smithi118 ceph-mon[29160]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.637+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/243705854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 msgr2=0x7f9a04105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.637+0000 7f9a0b0d0700 1 --2- 172.21.15.118:0/243705854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04105a40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009910 tx=0x7f99f8009c20 comp rx=0 tx=0).stop 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.637+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/243705854 shutdown_connections 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 --2- 172.21.15.118:0/243705854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04105a40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/243705854 >> 172.21.15.118:0/243705854 conn(0x7f9a040fb0e0 msgr2=0x7f9a040fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/243705854 shutdown_connections 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/243705854 wait complete. 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 Processor -- start 2024-03-12T03:18:23.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 -- start start 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04194f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a04195440 0x7f9a041997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a04195a60 con 0x7f9a04105660 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a0b0d0700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a04195bd0 con 0x7f9a04195440 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a08e6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04194f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a08e6c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04194f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:54020/0 (socket says 172.21.15.118:54020) 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.638+0000 7f9a08e6c700 1 -- 172.21.15.118:0/2034208277 learned_addr learned my addr 172.21.15.118:0/2034208277 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a08e6c700 1 -- 172.21.15.118:0/2034208277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a04195440 msgr2=0x7f9a041997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a08e6c700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a04195440 0x7f9a041997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a08e6c700 1 -- 172.21.15.118:0/2034208277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80095f0 con 0x7f9a04105660 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a08e6c700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04194f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f99f8005b40 tx=0x7f99f800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a01ffb700 1 -- 172.21.15.118:0/2034208277 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99f801d070 con 0x7f9a04105660 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a01ffb700 1 -- 172.21.15.118:0/2034208277 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99f8003d50 con 0x7f9a04105660 2024-03-12T03:18:23.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a04199df0 con 0x7f9a04105660 2024-03-12T03:18:23.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a01ffb700 1 -- 172.21.15.118:0/2034208277 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99f8017440 con 0x7f9a04105660 2024-03-12T03:18:23.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.639+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0419a3c0 con 0x7f9a04105660 2024-03-12T03:18:23.645 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.640+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a04100cd0 con 0x7f9a04105660 2024-03-12T03:18:23.645 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.641+0000 7f9a01ffb700 1 -- 172.21.15.118:0/2034208277 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) v1 ==== 90470+0+0 (secure 0 0 0) 0x7f99f8003890 con 0x7f9a04105660 2024-03-12T03:18:23.645 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.641+0000 7f9a01ffb700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f99f406c950 0x7f99f406ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:23.645 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.641+0000 7f9a01ffb700 1 -- 172.21.15.118:0/2034208277 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f99f808bda0 con 0x7f9a04105660 2024-03-12T03:18:23.645 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.641+0000 7f9a03fff700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f99f406c950 0x7f99f406ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:23.645 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.641+0000 7f9a03fff700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f99f406c950 0x7f99f406ee10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098e0 tx=0x7f99f0006dd0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:23.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.646+0000 7f9a01ffb700 1 -- 172.21.15.118:0/2034208277 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f99f8057510 con 0x7f9a04105660 2024-03-12T03:18:23.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:23 smithi177 ceph-mon[37140]: Reconfiguring mon.smithi118 (unknown last config time)... 2024-03-12T03:18:23.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:23 smithi177 ceph-mon[37140]: Reconfiguring daemon mon.smithi118 on smithi118 2024-03-12T03:18:23.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:23 smithi177 ceph-mon[37140]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:23.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.847+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9a041962b0 con 0x7f9a04105660 2024-03-12T03:18:23.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.848+0000 7f9a01ffb700 1 -- 172.21.15.118:0/2034208277 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f99f8026090 con 0x7f9a04105660 2024-03-12T03:18:23.851 INFO:teuthology.orchestra.run.smithi118.stdout:# minimal ceph.conf for 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:23.851 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-03-12T03:18:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T03:18:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: mon_host = [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] 2024-03-12T03:18:23.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f99f406c950 msgr2=0x7f99f406ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f99f406c950 0x7f99f406ee10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f99f00098e0 tx=0x7f99f0006dd0 comp rx=0 tx=0).stop 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 msgr2=0x7f9a04194f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04194f00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f99f8005b40 tx=0x7f99f800bdc0 comp rx=0 tx=0).stop 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 shutdown_connections 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f99f406c950 0x7f99f406ee10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9a04105660 0x7f9a04194f00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 --2- 172.21.15.118:0/2034208277 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9a04195440 0x7f9a041997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 >> 172.21.15.118:0/2034208277 conn(0x7f9a040fb0e0 msgr2=0x7f9a040fb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 shutdown_connections 2024-03-12T03:18:23.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:23.850+0000 7f9a0b0d0700 1 -- 172.21.15.118:0/2034208277 wait complete. 2024-03-12T03:18:24.669 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:24 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:24.669 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:24 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2034208277' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:24.669 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:24 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:24.669 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:24 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:24.669 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:24 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi118.zzzuvl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:18:24.670 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:24 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:18:24.670 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:24 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:24.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:24 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:24.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:24 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2034208277' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:24.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:24 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:24.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:24 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:24.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:24 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi118.zzzuvl", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:18:24.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:24 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:18:24.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:24 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:24.973 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T03:18:24.974 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:18:24.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:18:25.003 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:18:25.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:18:25.072 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:18:25.072 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:18:25.107 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:18:25.108 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:18:25.177 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T03:18:25.177 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:18:25.178 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:18:25.193 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:18:25.193 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-12 03:17:37.625574566 +0000 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-12 03:08:01.912833427 +0000 2024-03-12T03:18:25.248 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-12 03:08:01.912833427 +0000 2024-03-12T03:18:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-12 03:08:01.912833427 +0000 2024-03-12T03:18:25.249 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:18:25.314 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-12T03:18:25.314 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-12T03:18:25.314 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 8.7697e-05 s, 5.8 MB/s 2024-03-12T03:18:25.315 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:18:25.372 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-12 03:17:37.626574544 +0000 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-12 03:08:02.232826553 +0000 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-12 03:08:02.232826553 +0000 2024-03-12T03:18:25.428 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-12 03:08:02.232826553 +0000 2024-03-12T03:18:25.428 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:18:25.497 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-12T03:18:25.497 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-12T03:18:25.497 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 8.5298e-05 s, 6.0 MB/s 2024-03-12T03:18:25.498 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:18:25.554 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-03-12T03:18:25.609 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:18:25.609 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:25.610 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-12T03:18:25.610 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:25.610 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:25.610 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-12 03:17:37.626574544 +0000 2024-03-12T03:18:25.610 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-12 03:08:02.574819206 +0000 2024-03-12T03:18:25.610 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-12 03:08:02.574819206 +0000 2024-03-12T03:18:25.610 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-12 03:08:02.574819206 +0000 2024-03-12T03:18:25.610 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:18:25.674 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-12T03:18:25.675 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-12T03:18:25.675 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 8.3923e-05 s, 6.1 MB/s 2024-03-12T03:18:25.676 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:18:25.732 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-12 03:17:37.626574544 +0000 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-12 03:08:02.896812290 +0000 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-12 03:08:02.896812290 +0000 2024-03-12T03:18:25.787 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-12 03:08:02.896812290 +0000 2024-03-12T03:18:25.788 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:18:25.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:25 smithi118 ceph-mon[29160]: Reconfiguring mgr.smithi118.zzzuvl (unknown last config time)... 2024-03-12T03:18:25.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:25 smithi118 ceph-mon[29160]: Reconfiguring daemon mgr.smithi118.zzzuvl on smithi118 2024-03-12T03:18:25.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:25 smithi118 ceph-mon[29160]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:25.835 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-12T03:18:25.835 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-12T03:18:25.835 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 9.1553e-05 s, 5.6 MB/s 2024-03-12T03:18:25.837 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:18:25.893 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:18:25.893 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:18:25.912 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:18:25.912 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-03-12T03:18:25.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:25 smithi177 ceph-mon[37140]: Reconfiguring mgr.smithi118.zzzuvl (unknown last config time)... 2024-03-12T03:18:25.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:25 smithi177 ceph-mon[37140]: Reconfiguring daemon mgr.smithi118.zzzuvl on smithi118 2024-03-12T03:18:25.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:25 smithi177 ceph-mon[37140]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-12 03:18:12.568451041 +0000 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-12 03:08:05.903405302 +0000 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-12 03:08:05.903405302 +0000 2024-03-12T03:18:25.970 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-12 03:08:05.903405302 +0000 2024-03-12T03:18:25.971 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:18:26.040 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-12T03:18:26.040 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-12T03:18:26.040 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000305509 s, 1.7 MB/s 2024-03-12T03:18:26.041 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:18:26.099 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-03-12T03:18:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:18:26.156 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:26.156 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-12T03:18:26.156 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:26.156 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:26.156 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-12 03:18:12.569451013 +0000 2024-03-12T03:18:26.156 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-12 03:08:06.240395933 +0000 2024-03-12T03:18:26.157 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-12 03:08:06.240395933 +0000 2024-03-12T03:18:26.157 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-12 03:08:06.240395933 +0000 2024-03-12T03:18:26.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:18:26.225 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-12T03:18:26.225 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-12T03:18:26.225 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000194487 s, 2.6 MB/s 2024-03-12T03:18:26.226 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:18:26.283 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-12 03:18:12.569451013 +0000 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-12 03:08:06.565386897 +0000 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-12 03:08:06.565386897 +0000 2024-03-12T03:18:26.339 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-12 03:08:06.565386897 +0000 2024-03-12T03:18:26.340 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:18:26.406 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-12T03:18:26.406 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-12T03:18:26.406 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000192798 s, 2.7 MB/s 2024-03-12T03:18:26.407 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:18:26.464 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-12 03:18:12.569451013 +0000 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-12 03:08:06.914377193 +0000 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-12 03:08:06.914377193 +0000 2024-03-12T03:18:26.520 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-12 03:08:06.914377193 +0000 2024-03-12T03:18:26.521 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:18:26.586 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-12T03:18:26.586 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-12T03:18:26.586 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000199694 s, 2.6 MB/s 2024-03-12T03:18:26.587 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:18:26.643 INFO:tasks.cephadm:Deploying osd.0 on smithi118 with /dev/vg_nvme/lv_4... 2024-03-12T03:18:26.643 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:18:27.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:27.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:27.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:27 smithi177 ceph-mon[37140]: Reconfiguring ceph-exporter.smithi118 (monmap changed)... 2024-03-12T03:18:27.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi118", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-12T03:18:27.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:27.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:27 smithi177 ceph-mon[37140]: Reconfiguring daemon ceph-exporter.smithi118 on smithi118 2024-03-12T03:18:27.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:27 smithi177 ceph-mon[37140]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:27.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:27.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:27.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:27 smithi118 ceph-mon[29160]: Reconfiguring ceph-exporter.smithi118 (monmap changed)... 2024-03-12T03:18:27.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi118", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-12T03:18:27.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:27.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:27 smithi118 ceph-mon[29160]: Reconfiguring daemon ceph-exporter.smithi118 on smithi118 2024-03-12T03:18:27.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:27 smithi118 ceph-mon[29160]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:27.611 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:18:28.686 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:28.686 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:28.686 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:28 smithi118 ceph-mon[29160]: Reconfiguring crash.smithi118 (monmap changed)... 2024-03-12T03:18:28.686 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi118", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-12T03:18:28.686 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:28.686 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:28 smithi118 ceph-mon[29160]: Reconfiguring daemon crash.smithi118 on smithi118 2024-03-12T03:18:28.921 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:28.921 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:28.921 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:28 smithi177 ceph-mon[37140]: Reconfiguring crash.smithi118 (monmap changed)... 2024-03-12T03:18:28.921 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi118", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-12T03:18:28.921 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:28.921 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:28 smithi177 ceph-mon[37140]: Reconfiguring daemon crash.smithi118 on smithi118 2024-03-12T03:18:29.944 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:29 smithi118 ceph-mon[29160]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:29.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:29 smithi177 ceph-mon[37140]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:30.524 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:18:30.524 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:18:30.524 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-12T03:18:30.524 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-12T03:18:30.524 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00838103 s, 1.3 GB/s 2024-03-12T03:18:30.524 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-12T03:18:31.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:30 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:31.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:30 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:31.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:30 smithi118 ceph-mon[29160]: Reconfiguring alertmanager.smithi118 (dependencies changed)... 2024-03-12T03:18:31.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:30 smithi118 ceph-mon[29160]: Reconfiguring daemon alertmanager.smithi118 on smithi118 2024-03-12T03:18:31.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:30 smithi118 ceph-mon[29160]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:31.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:30 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:31.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:30 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:31.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:30 smithi177 ceph-mon[37140]: Reconfiguring alertmanager.smithi118 (dependencies changed)... 2024-03-12T03:18:31.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:30 smithi177 ceph-mon[37140]: Reconfiguring daemon alertmanager.smithi118 on smithi118 2024-03-12T03:18:31.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:30 smithi177 ceph-mon[37140]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:32.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_4 2024-03-12T03:18:33.564 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:18:33.687 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:33 smithi118 ceph-mon[29160]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:33.687 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:33 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:33.927 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:33 smithi177 ceph-mon[37140]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:33.927 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:33 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:34.983 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:34 smithi118 ceph-mon[29160]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:35.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:34 smithi177 ceph-mon[37140]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:35.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.858+0000 7f30ecf5b700 1 -- 172.21.15.118:0/285008047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e80fed40 msgr2=0x7f30e80ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:35.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.858+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/285008047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e80fed40 0x7f30e80ff160 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f30dc009910 tx=0x7f30dc009c20 comp rx=0 tx=0).stop 2024-03-12T03:18:35.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.858+0000 7f30ecf5b700 1 -- 172.21.15.118:0/285008047 shutdown_connections 2024-03-12T03:18:35.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.858+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/285008047 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30e8101d20 0x7f30e8102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:35.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.858+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/285008047 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e80fed40 0x7f30e80ff160 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:35.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.858+0000 7f30ecf5b700 1 -- 172.21.15.118:0/285008047 >> 172.21.15.118:0/285008047 conn(0x7f30e80fa6a0 msgr2=0x7f30e80fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:18:35.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.858+0000 7f30ecf5b700 1 -- 172.21.15.118:0/285008047 shutdown_connections 2024-03-12T03:18:35.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.859+0000 7f30ecf5b700 1 -- 172.21.15.118:0/285008047 wait complete. 2024-03-12T03:18:35.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.859+0000 7f30ecf5b700 1 Processor -- start 2024-03-12T03:18:35.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.859+0000 7f30ecf5b700 1 -- start start 2024-03-12T03:18:35.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.859+0000 7f30ecf5b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30e80fed40 0x7f30e81964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:35.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.859+0000 7f30ecf5b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e8101d20 0x7f30e8196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.859+0000 7f30ecf5b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e8197050 con 0x7f30e8101d20 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.859+0000 7f30ecf5b700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e8197190 con 0x7f30e80fed40 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e5d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e8101d20 0x7f30e8196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e5d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e8101d20 0x7f30e8196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48594/0 (socket says 172.21.15.118:48594) 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e5d9b700 1 -- 172.21.15.118:0/3587159974 learned_addr learned my addr 172.21.15.118:0/3587159974 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e659c700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30e80fed40 0x7f30e81964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e5d9b700 1 -- 172.21.15.118:0/3587159974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30e80fed40 msgr2=0x7f30e81964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e5d9b700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30e80fed40 0x7f30e81964f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e5d9b700 1 -- 172.21.15.118:0/3587159974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc0095f0 con 0x7f30e8101d20 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30e5d9b700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e8101d20 0x7f30e8196a30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c8a0 tx=0x7f30d800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30d77fe700 1 -- 172.21.15.118:0/3587159974 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d80078e0 con 0x7f30e8101d20 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30d77fe700 1 -- 172.21.15.118:0/3587159974 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30d800f450 con 0x7f30e8101d20 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e819bba0 con 0x7f30e8101d20 2024-03-12T03:18:35.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30d77fe700 1 -- 172.21.15.118:0/3587159974 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30d800e5e0 con 0x7f30e8101d20 2024-03-12T03:18:35.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.860+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e819c170 con 0x7f30e8101d20 2024-03-12T03:18:35.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.861+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e8190600 con 0x7f30e8101d20 2024-03-12T03:18:35.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.862+0000 7f30d77fe700 1 -- 172.21.15.118:0/3587159974 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) v1 ==== 90470+0+0 (secure 0 0 0) 0x7f30d8007a40 con 0x7f30e8101d20 2024-03-12T03:18:35.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.862+0000 7f30d77fe700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f30d006c800 0x7f30d006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:18:35.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.863+0000 7f30d77fe700 1 -- 172.21.15.118:0/3587159974 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f30d808ab10 con 0x7f30e8101d20 2024-03-12T03:18:35.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.863+0000 7f30e659c700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f30d006c800 0x7f30d006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:18:35.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.863+0000 7f30e659c700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f30d006c800 0x7f30d006ecc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f30dc000c00 tx=0x7f30dc005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:18:35.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:35.867+0000 7f30d77fe700 1 -- 172.21.15.118:0/3587159974 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f30d8059c90 con 0x7f30e8101d20 2024-03-12T03:18:36.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:18:36.089+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f30e80611f0 con 0x7f30d006c800 2024-03-12T03:18:36.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:36 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:18:36.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:36 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:18:36.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:36 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:36 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:36 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:18:36.460 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:36 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:37.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:37 smithi118 ceph-mon[29160]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:18:37.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:37 smithi118 ceph-mon[29160]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:37.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:37.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:37.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:37 smithi118 ceph-mon[29160]: Reconfiguring grafana.smithi118 (dependencies changed)... 2024-03-12T03:18:37.861 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:37 smithi118 ceph-mon[29160]: Reconfiguring daemon grafana.smithi118 on smithi118 2024-03-12T03:18:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:37 smithi177 ceph-mon[37140]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:18:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:37 smithi177 ceph-mon[37140]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:37 smithi177 ceph-mon[37140]: Reconfiguring grafana.smithi118 (dependencies changed)... 2024-03-12T03:18:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:37 smithi177 ceph-mon[37140]: Reconfiguring daemon grafana.smithi118 on smithi118 2024-03-12T03:18:39.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:39 smithi177 ceph-mon[37140]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:40.047 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:39 smithi118 ceph-mon[29160]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:42.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:41 smithi118 ceph-mon[29160]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:42.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:41 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:42.027 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:41 smithi177 ceph-mon[37140]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:42.028 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:41 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:43.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:42 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:43.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:42 smithi118 ceph-mon[29160]: Reconfiguring prometheus.smithi118 (dependencies changed)... 2024-03-12T03:18:43.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:42 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:43.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:42 smithi177 ceph-mon[37140]: Reconfiguring prometheus.smithi118 (dependencies changed)... 2024-03-12T03:18:44.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:43 smithi118 ceph-mon[29160]: Reconfiguring daemon prometheus.smithi118 on smithi118 2024-03-12T03:18:44.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:43 smithi118 ceph-mon[29160]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:44.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:43 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2712819843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96b74c65-8c14-4178-9e8d-b981032baa7d"}]: dispatch 2024-03-12T03:18:44.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:43 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2712819843' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96b74c65-8c14-4178-9e8d-b981032baa7d"}]': finished 2024-03-12T03:18:44.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:43 smithi118 ceph-mon[29160]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:18:44.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:18:44.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:43 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/945915141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:18:44.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:43 smithi177 ceph-mon[37140]: Reconfiguring daemon prometheus.smithi118 on smithi118 2024-03-12T03:18:44.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:43 smithi177 ceph-mon[37140]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:44.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:43 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2712819843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96b74c65-8c14-4178-9e8d-b981032baa7d"}]: dispatch 2024-03-12T03:18:44.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:43 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2712819843' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96b74c65-8c14-4178-9e8d-b981032baa7d"}]': finished 2024-03-12T03:18:44.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:43 smithi177 ceph-mon[37140]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:18:44.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:18:44.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:43 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/945915141' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:18:45.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:44 smithi177 ceph-mon[37140]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:45.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:44 smithi118 ceph-mon[29160]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:47.769 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:47 smithi118 ceph-mon[29160]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:47.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:47 smithi177 ceph-mon[37140]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:48.813 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:48.947 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:18:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:49 smithi118 ceph-mon[29160]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:49.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:49 smithi177 ceph-mon[37140]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:51 smithi118 ceph-mon[29160]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:51.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:51 smithi177 ceph-mon[37140]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:53.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:53 smithi177 ceph-mon[37140]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:53.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:53.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:53.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:53 smithi177 ceph-mon[37140]: Reconfiguring ceph-exporter.smithi177 (monmap changed)... 2024-03-12T03:18:53.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi177", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-12T03:18:53.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:53.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:53 smithi177 ceph-mon[37140]: Reconfiguring daemon ceph-exporter.smithi177 on smithi177 2024-03-12T03:18:54.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:53 smithi118 ceph-mon[29160]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:54.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:54.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:54.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:53 smithi118 ceph-mon[29160]: Reconfiguring ceph-exporter.smithi177 (monmap changed)... 2024-03-12T03:18:54.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi177", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-03-12T03:18:54.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:54.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:53 smithi118 ceph-mon[29160]: Reconfiguring daemon ceph-exporter.smithi177 on smithi177 2024-03-12T03:18:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:18:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: Deploying daemon osd.0 on smithi118 2024-03-12T03:18:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: Reconfiguring crash.smithi177 (monmap changed)... 2024-03-12T03:18:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi177", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-12T03:18:55.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:55.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: Reconfiguring daemon crash.smithi177 on smithi177 2024-03-12T03:18:55.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:54 smithi118 ceph-mon[29160]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:55.178 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: Deploying daemon osd.0 on smithi118 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: Reconfiguring crash.smithi177 (monmap changed)... 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi177", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: Reconfiguring daemon crash.smithi177 on smithi177 2024-03-12T03:18:55.179 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:54 smithi177 ceph-mon[37140]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:56.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:56 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:56.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:56 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:56.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:56 smithi177 ceph-mon[37140]: Reconfiguring mgr.smithi177.tmmsuz (monmap changed)... 2024-03-12T03:18:56.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:56 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi177.tmmsuz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:18:56.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:56 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:18:56.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:56 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:56.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:56 smithi177 ceph-mon[37140]: Reconfiguring daemon mgr.smithi177.tmmsuz on smithi177 2024-03-12T03:18:56.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:56.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:56.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:56 smithi118 ceph-mon[29160]: Reconfiguring mgr.smithi177.tmmsuz (monmap changed)... 2024-03-12T03:18:56.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi177.tmmsuz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:18:56.709 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:18:56.709 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:56 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:56.709 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:56 smithi118 ceph-mon[29160]: Reconfiguring daemon mgr.smithi177.tmmsuz on smithi177 2024-03-12T03:18:57.556 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:57 smithi177 ceph-mon[37140]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:57.556 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:57.556 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:57.557 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:18:57.557 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:18:57.557 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:57.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:57 smithi118 ceph-mon[29160]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:57.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:57.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:57.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:18:57.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:18:57.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:18:58.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: Reconfiguring mon.smithi177 (monmap changed)... 2024-03-12T03:18:58.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: Reconfiguring daemon mon.smithi177 on smithi177 2024-03-12T03:18:58.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi118.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.562 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:58.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: Reconfiguring mon.smithi177 (monmap changed)... 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: Reconfiguring daemon mon.smithi177 on smithi177 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi118.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:58.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:18:59.693 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi118.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:59.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:18:59 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:59.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:18:59.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi118.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:18:59.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:18:59.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:18:59.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:18:59.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:18:59.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:18:59.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:18:59.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:18:59 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:01.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:01 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:01.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:01 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:01.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:01 smithi118 ceph-mon[29160]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:01.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:01 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:01.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:01 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:01.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:01 smithi177 ceph-mon[37140]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:03.667 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 0 on host 'smithi118' 2024-03-12T03:19:03.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.664+0000 7f30d77fe700 1 -- 172.21.15.118:0/3587159974 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f30e80611f0 con 0x7f30d006c800 2024-03-12T03:19:03.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f30d006c800 msgr2=0x7f30d006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:03.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f30d006c800 0x7f30d006ecc0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f30dc000c00 tx=0x7f30dc005960 comp rx=0 tx=0).stop 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e8101d20 msgr2=0x7f30e8196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e8101d20 0x7f30e8196a30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f30d800c8a0 tx=0x7f30d800cbb0 comp rx=0 tx=0).stop 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 shutdown_connections 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f30d006c800 0x7f30d006ecc0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f30e80fed40 0x7f30e81964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 --2- 172.21.15.118:0/3587159974 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f30e8101d20 0x7f30e8196a30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 >> 172.21.15.118:0/3587159974 conn(0x7f30e80fa6a0 msgr2=0x7f30e8105370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 shutdown_connections 2024-03-12T03:19:03.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:03.666+0000 7f30ecf5b700 1 -- 172.21.15.118:0/3587159974 wait complete. 2024-03-12T03:19:03.859 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:03 smithi118 ceph-mon[29160]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:03.859 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:03 smithi118 ceph-mon[29160]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:19:03.859 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:03.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:03 smithi177 ceph-mon[37140]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:03.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:03 smithi177 ceph-mon[37140]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:19:03.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:04.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:04 smithi118 ceph-mon[29160]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:19:04.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:04 smithi118 ceph-mon[29160]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:19:04.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:04.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:04 smithi118 ceph-mon[29160]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-12T03:19:04.924 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:04.924 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:04.944 DEBUG:teuthology.orchestra.run.smithi118:osd.0> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T03:19:04.947 INFO:tasks.cephadm:Deploying osd.1 on smithi118 with /dev/vg_nvme/lv_3... 2024-03-12T03:19:04.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:19:04.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:04 smithi177 ceph-mon[37140]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:19:04.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:04 smithi177 ceph-mon[37140]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:19:04.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:04.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:04 smithi177 ceph-mon[37140]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-12T03:19:04.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:04.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:05.701 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:19:05.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:05 smithi118 ceph-mon[29160]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:05.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:05 smithi118 ceph-mon[29160]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-12T03:19:05.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:05 smithi118 ceph-mon[29160]: osdmap e8: 1 total, 0 up, 1 in 2024-03-12T03:19:05.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:05 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:05.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:05 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:05.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:05 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:05.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:05 smithi177 ceph-mon[37140]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:05.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:05 smithi177 ceph-mon[37140]: from='osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-12T03:19:05.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:05 smithi177 ceph-mon[37140]: osdmap e8: 1 total, 0 up, 1 in 2024-03-12T03:19:05.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:05 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:05.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:05 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:05.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:05 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:06.856 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:06 smithi118 ceph-mon[29160]: purged_snaps scrub starts 2024-03-12T03:19:06.856 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:06 smithi118 ceph-mon[29160]: purged_snaps scrub ok 2024-03-12T03:19:06.856 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:06 smithi118 ceph-mon[29160]: osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] boot 2024-03-12T03:19:06.856 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:06 smithi118 ceph-mon[29160]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:19:06.856 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:06.857 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:06.857 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:06.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:06 smithi177 ceph-mon[37140]: purged_snaps scrub starts 2024-03-12T03:19:06.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:06 smithi177 ceph-mon[37140]: purged_snaps scrub ok 2024-03-12T03:19:06.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:06 smithi177 ceph-mon[37140]: osd.0 [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] boot 2024-03-12T03:19:06.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:06 smithi177 ceph-mon[37140]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:19:06.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:19:06.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:06.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:07.729 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:19:07.729 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:19:07.729 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-12T03:19:07.729 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-12T03:19:07.729 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00862633 s, 1.2 GB/s 2024-03-12T03:19:07.729 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-12T03:19:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:07 smithi118 ceph-mon[29160]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:07 smithi118 ceph-mon[29160]: osdmap e10: 1 total, 1 up, 1 in 2024-03-12T03:19:08.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:07 smithi177 ceph-mon[37140]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:19:08.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:07 smithi177 ceph-mon[37140]: osdmap e10: 1 total, 1 up, 1 in 2024-03-12T03:19:08.702 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_3 2024-03-12T03:19:09.700 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:19:09.822 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:09 smithi118 ceph-mon[29160]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:10.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:09 smithi177 ceph-mon[37140]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:11.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.850+0000 7f509df58700 1 -- 172.21.15.118:0/3305964924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5098102060 msgr2=0x7f5098102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.850+0000 7f509df58700 1 --2- 172.21.15.118:0/3305964924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5098102060 0x7f5098102480 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5088009910 tx=0x7f5088009c20 comp rx=0 tx=0).stop 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.850+0000 7f509df58700 1 -- 172.21.15.118:0/3305964924 shutdown_connections 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.850+0000 7f509df58700 1 --2- 172.21.15.118:0/3305964924 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f50980feae0 0x7f50980fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.850+0000 7f509df58700 1 --2- 172.21.15.118:0/3305964924 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5098102060 0x7f5098102480 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.850+0000 7f509df58700 1 -- 172.21.15.118:0/3305964924 >> 172.21.15.118:0/3305964924 conn(0x7f50980fa6a0 msgr2=0x7f50980fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 -- 172.21.15.118:0/3305964924 shutdown_connections 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 -- 172.21.15.118:0/3305964924 wait complete. 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 Processor -- start 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 -- start start 2024-03-12T03:19:11.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50980feae0 0x7f5098196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098102060 0x7f5098196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50981970e0 con 0x7f50980feae0 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.851+0000 7f509df58700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5098197220 con 0x7f5098102060 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f50977fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50980feae0 0x7f5098196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f50977fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50980feae0 0x7f5098196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47856/0 (socket says 172.21.15.118:47856) 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f50977fe700 1 -- 172.21.15.118:0/3215286984 learned_addr learned my addr 172.21.15.118:0/3215286984 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f5096ffd700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098102060 0x7f5098196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f50977fe700 1 -- 172.21.15.118:0/3215286984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098102060 msgr2=0x7f5098196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f50977fe700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098102060 0x7f5098196ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f50977fe700 1 -- 172.21.15.118:0/3215286984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50880095f0 con 0x7f50980feae0 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f50977fe700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50980feae0 0x7f5098196580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f508800c010 tx=0x7f5088005e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f5094ff9700 1 -- 172.21.15.118:0/3215286984 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f508801d070 con 0x7f50980feae0 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509819bbd0 con 0x7f50980feae0 2024-03-12T03:19:11.856 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f5094ff9700 1 -- 172.21.15.118:0/3215286984 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50880052a0 con 0x7f50980feae0 2024-03-12T03:19:11.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f5094ff9700 1 -- 172.21.15.118:0/3215286984 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5088017400 con 0x7f50980feae0 2024-03-12T03:19:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.852+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509819c0c0 con 0x7f50980feae0 2024-03-12T03:19:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.854+0000 7f5094ff9700 1 -- 172.21.15.118:0/3215286984 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) v1 ==== 90470+0+0 (secure 0 0 0) 0x7f5088003710 con 0x7f50980feae0 2024-03-12T03:19:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.854+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5098190630 con 0x7f50980feae0 2024-03-12T03:19:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.854+0000 7f5094ff9700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f508006c800 0x7f508006ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:19:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.855+0000 7f5094ff9700 1 -- 172.21.15.118:0/3215286984 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f508808c1b0 con 0x7f50980feae0 2024-03-12T03:19:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.855+0000 7f5096ffd700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f508006c800 0x7f508006ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:19:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.855+0000 7f5096ffd700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f508006c800 0x7f508006ecc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f508c009c60 tx=0x7f508c009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:19:11.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:11.860+0000 7f5094ff9700 1 -- 172.21.15.118:0/3215286984 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f508805b0d0 con 0x7f50980feae0 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:12.005 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:12.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:12.077+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f50980611f0 con 0x7f508006c800 2024-03-12T03:19:12.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:12 smithi177 ceph-mon[37140]: Detected new or changed devices on smithi118 2024-03-12T03:19:12.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:12 smithi177 ceph-mon[37140]: Adjusting osd_memory_target on smithi118 to 11840M 2024-03-12T03:19:12.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:12 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:19:12.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:12 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:19:12.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:12 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:13.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:12 smithi118 ceph-mon[29160]: Detected new or changed devices on smithi118 2024-03-12T03:19:13.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:12 smithi118 ceph-mon[29160]: Adjusting osd_memory_target on smithi118 to 11840M 2024-03-12T03:19:13.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:12 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:19:13.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:12 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:19:13.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:12 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:13.944 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:13 smithi118 ceph-mon[29160]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:13.944 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:13 smithi118 ceph-mon[29160]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:13.944 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:13 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:13.944 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:13 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:14.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:13 smithi177 ceph-mon[37140]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:14.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:13 smithi177 ceph-mon[37140]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:14.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:13 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:14.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:13 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:15.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:14 smithi118 ceph-mon[29160]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:15.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:14 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1430409259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c911034-856e-43e7-b267-2a6c63cf46fd"}]: dispatch 2024-03-12T03:19:15.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:14 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1430409259' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c911034-856e-43e7-b267-2a6c63cf46fd"}]': finished 2024-03-12T03:19:15.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:14 smithi118 ceph-mon[29160]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:19:15.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:14 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:15.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:14 smithi177 ceph-mon[37140]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:15.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:14 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1430409259' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3c911034-856e-43e7-b267-2a6c63cf46fd"}]: dispatch 2024-03-12T03:19:15.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:14 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1430409259' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3c911034-856e-43e7-b267-2a6c63cf46fd"}]': finished 2024-03-12T03:19:15.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:14 smithi177 ceph-mon[37140]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:19:15.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:14 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:16.141 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:15 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3986707725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:19:16.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:15 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3986707725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:19:16.938 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:16 smithi118 ceph-mon[29160]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:17.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:16 smithi177 ceph-mon[37140]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:18.512 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:18.512 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:18.512 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:18.512 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:18.512 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:18.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:18.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:18.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:18.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:18.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:19.502 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:19 smithi118 ceph-mon[29160]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:19.502 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:19.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:19 smithi177 ceph-mon[37140]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:19.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:21.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:21 smithi177 ceph-mon[37140]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:22.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:21 smithi118 ceph-mon[29160]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:22.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:19:22.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:22.964 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:19:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:23.918 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:23 smithi118 ceph-mon[29160]: Deploying daemon osd.1 on smithi118 2024-03-12T03:19:23.918 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:23 smithi118 ceph-mon[29160]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:23.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:23 smithi177 ceph-mon[37140]: Deploying daemon osd.1 on smithi118 2024-03-12T03:19:23.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:23 smithi177 ceph-mon[37140]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:25.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:24 smithi118 ceph-mon[29160]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:25.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:24 smithi177 ceph-mon[37140]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:27.891 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:27 smithi118 ceph-mon[29160]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:27.891 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:27.891 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:27.891 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:27.917 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:27 smithi177 ceph-mon[37140]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:27.917 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:27.917 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:27.918 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:29.895 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:29 smithi118 ceph-mon[29160]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:29.895 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:29 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:29.895 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:29 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:29.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:29 smithi177 ceph-mon[37140]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:29.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:29 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:29.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:29 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:30.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.528+0000 7f5094ff9700 1 -- 172.21.15.118:0/3215286984 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f50980611f0 con 0x7f508006c800 2024-03-12T03:19:30.531 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 1 on host 'smithi118' 2024-03-12T03:19:30.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f508006c800 msgr2=0x7f508006ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f508006c800 0x7f508006ecc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f508c009c60 tx=0x7f508c009260 comp rx=0 tx=0).stop 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50980feae0 msgr2=0x7f5098196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50980feae0 0x7f5098196580 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f508800c010 tx=0x7f5088005e30 comp rx=0 tx=0).stop 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 shutdown_connections 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f508006c800 0x7f508006ecc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f50980feae0 0x7f5098196580 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 --2- 172.21.15.118:0/3215286984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5098102060 0x7f5098196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 >> 172.21.15.118:0/3215286984 conn(0x7f50980fa6a0 msgr2=0x7f5098105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 shutdown_connections 2024-03-12T03:19:30.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:30.530+0000 7f509df58700 1 -- 172.21.15.118:0/3215286984 wait complete. 2024-03-12T03:19:31.208 DEBUG:teuthology.orchestra.run.smithi118:osd.1> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T03:19:31.210 INFO:tasks.cephadm:Deploying osd.2 on smithi118 with /dev/vg_nvme/lv_2... 2024-03-12T03:19:31.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:19:31.686 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:19:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:31 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:31 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:31 smithi118 ceph-mon[29160]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:31 smithi118 ceph-mon[29160]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:19:31.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:31 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:31.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:31 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:31.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:31 smithi177 ceph-mon[37140]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:31.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:31 smithi177 ceph-mon[37140]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:19:32.739 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:32 smithi118 ceph-mon[29160]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:19:32.739 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:32 smithi118 ceph-mon[29160]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:19:32.739 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:32 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:32.739 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:32 smithi118 ceph-mon[29160]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-12T03:19:32.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:19:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:19:32.508+0000 7f3508d45700 -1 osd.1 0 waiting for initial osdmap 2024-03-12T03:19:32.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:19:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:19:32.513+0000 7f3501b34700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:19:32.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:32 smithi177 ceph-mon[37140]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:19:32.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:32 smithi177 ceph-mon[37140]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:19:32.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:32 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:32.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:32 smithi177 ceph-mon[37140]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-12T03:19:33.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:33 smithi118 ceph-mon[29160]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-12T03:19:33.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:33 smithi118 ceph-mon[29160]: osdmap e13: 2 total, 1 up, 2 in 2024-03-12T03:19:33.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:33 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:33.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:33 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:33.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:33 smithi118 ceph-mon[29160]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:33.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:33 smithi177 ceph-mon[37140]: from='osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-12T03:19:33.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:33 smithi177 ceph-mon[37140]: osdmap e13: 2 total, 1 up, 2 in 2024-03-12T03:19:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:33 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:33 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:33 smithi177 ceph-mon[37140]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:19:34.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:34 smithi118 ceph-mon[29160]: purged_snaps scrub starts 2024-03-12T03:19:34.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:34 smithi118 ceph-mon[29160]: purged_snaps scrub ok 2024-03-12T03:19:34.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:34 smithi118 ceph-mon[29160]: osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] boot 2024-03-12T03:19:34.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:34 smithi118 ceph-mon[29160]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:19:34.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:34 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:34.708 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:34 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:34.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:34 smithi177 ceph-mon[37140]: purged_snaps scrub starts 2024-03-12T03:19:34.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:34 smithi177 ceph-mon[37140]: purged_snaps scrub ok 2024-03-12T03:19:34.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:34 smithi177 ceph-mon[37140]: osd.1 [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] boot 2024-03-12T03:19:34.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:34 smithi177 ceph-mon[37140]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:19:34.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:34 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:19:34.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:34 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:35.435 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:19:35.435 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:19:35.435 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-12T03:19:35.435 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-12T03:19:35.435 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00877559 s, 1.2 GB/s 2024-03-12T03:19:35.435 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-12T03:19:35.764 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:35 smithi118 ceph-mon[29160]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:35.764 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:35 smithi118 ceph-mon[29160]: osdmap e15: 2 total, 2 up, 2 in 2024-03-12T03:19:35.765 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:35 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:35.765 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:35 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:35.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:35 smithi177 ceph-mon[37140]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:35.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:35 smithi177 ceph-mon[37140]: osdmap e15: 2 total, 2 up, 2 in 2024-03-12T03:19:35.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:35 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:35.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:35 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:36.072 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_2 2024-03-12T03:19:37.233 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:19:37.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:37 smithi118 ceph-mon[29160]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:37 smithi177 ceph-mon[37140]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:39.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 -- 172.21.15.118:0/3532894142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d600ff9c0 msgr2=0x7f9d600ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 --2- 172.21.15.118:0/3532894142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d600ff9c0 0x7f9d600ffde0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009910 tx=0x7f9d50009c20 comp rx=0 tx=0).stop 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 -- 172.21.15.118:0/3532894142 shutdown_connections 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 --2- 172.21.15.118:0/3532894142 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d60100bc0 0x7f9d60101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 --2- 172.21.15.118:0/3532894142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d600ff9c0 0x7f9d600ffde0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 -- 172.21.15.118:0/3532894142 >> 172.21.15.118:0/3532894142 conn(0x7f9d600faf80 msgr2=0x7f9d600fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 -- 172.21.15.118:0/3532894142 shutdown_connections 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.812+0000 7f9d659eb700 1 -- 172.21.15.118:0/3532894142 wait complete. 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d659eb700 1 Processor -- start 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d659eb700 1 -- start start 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d659eb700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d600ff9c0 0x7f9d60195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d659eb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d60100bc0 0x7f9d601959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d659eb700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d60195fc0 con 0x7f9d60100bc0 2024-03-12T03:19:39.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d659eb700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d60196100 con 0x7f9d600ff9c0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d5e7fc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d60100bc0 0x7f9d601959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d5e7fc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d60100bc0 0x7f9d601959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:34308/0 (socket says 172.21.15.118:34308) 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d5e7fc700 1 -- 172.21.15.118:0/2009281976 learned_addr learned my addr 172.21.15.118:0/2009281976 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.813+0000 7f9d5effd700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d600ff9c0 0x7f9d60195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d5e7fc700 1 -- 172.21.15.118:0/2009281976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d600ff9c0 msgr2=0x7f9d60195460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d5e7fc700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d600ff9c0 0x7f9d60195460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d5e7fc700 1 -- 172.21.15.118:0/2009281976 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d500095f0 con 0x7f9d60100bc0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d5e7fc700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d60100bc0 0x7f9d601959a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9d5400eab0 tx=0x7f9d5400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d649e9700 1 -- 172.21.15.118:0/2009281976 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d54013070 con 0x7f9d60100bc0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d60068fb0 con 0x7f9d60100bc0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d649e9700 1 -- 172.21.15.118:0/2009281976 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d54017410 con 0x7f9d60100bc0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d649e9700 1 -- 172.21.15.118:0/2009281976 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d540165e0 con 0x7f9d60100bc0 2024-03-12T03:19:39.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.814+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d60069580 con 0x7f9d60100bc0 2024-03-12T03:19:39.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.815+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6004ea80 con 0x7f9d60100bc0 2024-03-12T03:19:39.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.816+0000 7f9d649e9700 1 -- 172.21.15.118:0/2009281976 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 18) v1 ==== 90470+0+0 (secure 0 0 0) 0x7f9d54017580 con 0x7f9d60100bc0 2024-03-12T03:19:39.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.816+0000 7f9d649e9700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9d4806c800 0x7f9d4806ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:19:39.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.816+0000 7f9d5effd700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9d4806c800 0x7f9d4806ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:19:39.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.816+0000 7f9d649e9700 1 -- 172.21.15.118:0/2009281976 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f9d5408c7b0 con 0x7f9d60100bc0 2024-03-12T03:19:39.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.816+0000 7f9d5effd700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9d4806c800 0x7f9d4806ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000b800 tx=0x7f9d50009f70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:19:39.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:39.821+0000 7f9d649e9700 1 -- 172.21.15.118:0/2009281976 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9d5405b5d0 con 0x7f9d60100bc0 2024-03-12T03:19:39.902 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:39.902 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:39.902 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:39.902 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:39.902 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:39.903 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:39.903 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:39.903 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:39.903 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:39.903 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:39 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:19:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:40.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:40.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:40.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:40.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:39 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:40.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:19:40.068+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d60069800 con 0x7f9d4806c800 2024-03-12T03:19:40.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:40 smithi177 ceph-mon[37140]: Detected new or changed devices on smithi118 2024-03-12T03:19:40.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:40 smithi177 ceph-mon[37140]: Adjusting osd_memory_target on smithi118 to 5920M 2024-03-12T03:19:40.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:40 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:19:40.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:40 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:19:40.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:40 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:40.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:40 smithi118 ceph-mon[29160]: Detected new or changed devices on smithi118 2024-03-12T03:19:40.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:40 smithi118 ceph-mon[29160]: Adjusting osd_memory_target on smithi118 to 5920M 2024-03-12T03:19:40.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:40 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:19:40.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:40 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:19:40.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:40 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:41.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:41 smithi118 ceph-mon[29160]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:41.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:41 smithi118 ceph-mon[29160]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:41.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:41 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:41.869 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:41 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:41.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:41 smithi177 ceph-mon[37140]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:19:41.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:41 smithi177 ceph-mon[37140]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:41.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:41 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:41.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:41 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:43.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:43 smithi118 ceph-mon[29160]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:43.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:43 smithi177 ceph-mon[37140]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:44.602 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:44 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/4240866752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a69006-8aa7-40ea-8410-5645691cba22"}]: dispatch 2024-03-12T03:19:44.602 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:44 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/4240866752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a69006-8aa7-40ea-8410-5645691cba22"}]': finished 2024-03-12T03:19:44.602 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:44 smithi118 ceph-mon[29160]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:19:44.602 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:44 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:19:44.603 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:44 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3509971189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:19:44.951 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:44 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/4240866752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4a69006-8aa7-40ea-8410-5645691cba22"}]: dispatch 2024-03-12T03:19:44.951 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:44 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/4240866752' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4a69006-8aa7-40ea-8410-5645691cba22"}]': finished 2024-03-12T03:19:44.951 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:44 smithi177 ceph-mon[37140]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:19:44.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:44 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:19:44.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:44 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3509971189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:19:45.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:45 smithi177 ceph-mon[37140]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:46.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:45 smithi118 ceph-mon[29160]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:46.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:46.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:46.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:46.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:19:46.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:47.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:47 smithi177 ceph-mon[37140]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:47.968 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:47 smithi118 ceph-mon[29160]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:48.731 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:48.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:19:49.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:49 smithi177 ceph-mon[37140]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:49.998 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:49 smithi118 ceph-mon[29160]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:51 smithi118 ceph-mon[29160]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:51 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:19:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:51 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:51.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:51 smithi177 ceph-mon[37140]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:51.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:51 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:19:51.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:51 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:19:52.740 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:52 smithi118 ceph-mon[29160]: Deploying daemon osd.2 on smithi118 2024-03-12T03:19:52.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:52 smithi177 ceph-mon[37140]: Deploying daemon osd.2 on smithi118 2024-03-12T03:19:53.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:53 smithi118 ceph-mon[29160]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:53.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:53 smithi177 ceph-mon[37140]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:54 smithi118 ceph-mon[29160]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:55.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:54 smithi177 ceph-mon[37140]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:57.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:57.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:57.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:57.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:57 smithi118 ceph-mon[29160]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:19:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:19:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:57 smithi177 ceph-mon[37140]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:59.704 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:19:59 smithi118 ceph-mon[29160]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:19:59.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:19:59 smithi177 ceph-mon[37140]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:20:00.178 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 2 on host 'smithi118' 2024-03-12T03:20:00.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.175+0000 7f9d649e9700 1 -- 172.21.15.118:0/2009281976 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9d60069800 con 0x7f9d4806c800 2024-03-12T03:20:00.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9d4806c800 msgr2=0x7f9d4806ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:00.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9d4806c800 0x7f9d4806ecc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000b800 tx=0x7f9d50009f70 comp rx=0 tx=0).stop 2024-03-12T03:20:00.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d60100bc0 msgr2=0x7f9d601959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d60100bc0 0x7f9d601959a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9d5400eab0 tx=0x7f9d5400edc0 comp rx=0 tx=0).stop 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 shutdown_connections 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9d4806c800 0x7f9d4806ecc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9d600ff9c0 0x7f9d60195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 --2- 172.21.15.118:0/2009281976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d60100bc0 0x7f9d601959a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 >> 172.21.15.118:0/2009281976 conn(0x7f9d600faf80 msgr2=0x7f9d60104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 shutdown_connections 2024-03-12T03:20:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:20:00.177+0000 7f9d659eb700 1 -- 172.21.15.118:0/2009281976 wait complete. 2024-03-12T03:20:00.774 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:00 smithi118 ceph-mon[29160]: overall HEALTH_OK 2024-03-12T03:20:00.774 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:00 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:00.774 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:00 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:00.774 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:00 smithi118 ceph-mon[29160]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:20:00.792 DEBUG:teuthology.orchestra.run.smithi118:osd.2> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T03:20:00.794 INFO:tasks.cephadm:Deploying osd.3 on smithi177 with /dev/vg_nvme/lv_4... 2024-03-12T03:20:00.795 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:20:00.845 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:00 smithi177 ceph-mon[37140]: overall HEALTH_OK 2024-03-12T03:20:00.845 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:00 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:00.845 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:00 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:00.845 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:00 smithi177 ceph-mon[37140]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:20:01.009 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi177/config 2024-03-12T03:20:01.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:01 smithi177 ceph-mon[37140]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:20:01.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:01 smithi177 ceph-mon[37140]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:20:01.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:01 smithi177 ceph-mon[37140]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:20:01.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:01 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:01.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:01 smithi177 ceph-mon[37140]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-12T03:20:01.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:01 smithi118 ceph-mon[29160]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:20:01.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:01 smithi118 ceph-mon[29160]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:20:01.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:01 smithi118 ceph-mon[29160]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:20:01.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:01 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:01.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:01 smithi118 ceph-mon[29160]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-12T03:20:02.395 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:20:02.160+0000 7f018aa23700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T03:20:02.395 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:20:02.167+0000 7f0184814700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:20:02.899 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:20:02.899 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:20:02.899 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-12T03:20:02.899 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-12T03:20:02.899 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112799 s, 930 MB/s 2024-03-12T03:20:02.899 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-12T03:20:03.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:03 smithi177 ceph-mon[37140]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-12T03:20:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:03 smithi177 ceph-mon[37140]: osdmap e18: 3 total, 2 up, 3 in 2024-03-12T03:20:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:03 smithi177 ceph-mon[37140]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:20:03.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-03-12T03:20:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:03 smithi118 ceph-mon[29160]: from='osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-12T03:20:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:03 smithi118 ceph-mon[29160]: osdmap e18: 3 total, 2 up, 3 in 2024-03-12T03:20:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:03.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:03 smithi118 ceph-mon[29160]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:20:03.695 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi177/config 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: purged_snaps scrub starts 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: purged_snaps scrub ok 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] boot 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:04.229 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:04.348 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: purged_snaps scrub starts 2024-03-12T03:20:04.348 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: purged_snaps scrub ok 2024-03-12T03:20:04.348 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: osd.2 [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] boot 2024-03-12T03:20:04.348 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:20:04.348 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:20:04.349 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:04.349 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:04.349 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:05.106 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- 172.21.15.177:0/3677478522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c100ab0 msgr2=0x7f7b5c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:05.106 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/3677478522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c100ab0 0x7f7b5c100ed0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009910 tx=0x7f7b50009c20 comp rx=0 tx=0).stop 2024-03-12T03:20:05.106 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- 172.21.15.177:0/3677478522 shutdown_connections 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/3677478522 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b5c101cb0 0x7f7b5c102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/3677478522 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c100ab0 0x7f7b5c100ed0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- 172.21.15.177:0/3677478522 >> 172.21.15.177:0/3677478522 conn(0x7f7b5c0fc090 msgr2=0x7f7b5c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- 172.21.15.177:0/3677478522 shutdown_connections 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- 172.21.15.177:0/3677478522 wait complete. 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 Processor -- start 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- start start 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b5c100ab0 0x7f7b5c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c101cb0 0x7f7b5c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c197070 con 0x7f7b5c100ab0 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.102+0000 7f7b62cd6700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5c1971b0 con 0x7f7b5c101cb0 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b5bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c101cb0 0x7f7b5c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:05.107 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b60a72700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b5c100ab0 0x7f7b5c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b5bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c101cb0 0x7f7b5c196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:50578/0 (socket says 172.21.15.177:50578) 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b5bfff700 1 -- 172.21.15.177:0/451455746 learned_addr learned my addr 172.21.15.177:0/451455746 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b5bfff700 1 -- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b5c100ab0 msgr2=0x7f7b5c196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b5bfff700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b5c100ab0 0x7f7b5c196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b5bfff700 1 -- 172.21.15.177:0/451455746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b500095f0 con 0x7f7b5c101cb0 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b5bfff700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c101cb0 0x7f7b5c196a50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800d8d0 tx=0x7f7b4800dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b480098a0 con 0x7f7b5c101cb0 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b48010460 con 0x7f7b5c101cb0 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5c19bbc0 con 0x7f7b5c101cb0 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b4800f5f0 con 0x7f7b5c101cb0 2024-03-12T03:20:05.108 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.103+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b5c19c190 con 0x7f7b5c101cb0 2024-03-12T03:20:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.104+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b5c190630 con 0x7f7b5c101cb0 2024-03-12T03:20:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.105+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 18) v1 ==== 90470+0+0 (secure 0 0 0) 0x7f7b48009a00 con 0x7f7b5c101cb0 2024-03-12T03:20:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.105+0000 7f7b59ffb700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7b4c06c800 0x7f7b4c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.105+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 2846+0+0 (secure 0 0 0) 0x7f7b4808aff0 con 0x7f7b5c101cb0 2024-03-12T03:20:05.109 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.106+0000 7f7b60a72700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7b4c06c800 0x7f7b4c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:05.110 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.106+0000 7f7b60a72700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7b4c06c800 0x7f7b4c06ecc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7b50000c00 tx=0x7f7b50005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:20:05.112 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.110+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7b48059bb0 con 0x7f7b5c101cb0 2024-03-12T03:20:05.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:05 smithi118 ceph-mon[29160]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:20:05.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:05 smithi118 ceph-mon[29160]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:05.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:05 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:05.313 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:05.310+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7b5c0611f0 con 0x7f7b4c06c800 2024-03-12T03:20:05.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:05 smithi177 ceph-mon[37140]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:20:05.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:05 smithi177 ceph-mon[37140]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:05.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:05 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:06.358 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:06.358 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:06 smithi177 ceph-mon[37140]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:20:06.358 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:06.358 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:06 smithi177 ceph-mon[37140]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:06.358 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:06.358 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:06.359 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:06 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:06.535 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:06.535 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:06 smithi118 ceph-mon[29160]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:20:06.535 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:20:06.535 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:06 smithi118 ceph-mon[29160]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:06.535 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:06.535 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:06.535 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:06 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:07.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:20:07.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:07.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67139]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:07.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67139]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:07.279 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: Detected new or changed devices on smithi118 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: Adjusting osd_memory_target on smithi118 to 3946M 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.280 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:07 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:07.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: osdmap e22: 3 total, 3 up, 3 in 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: Detected new or changed devices on smithi118 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: Adjusting osd_memory_target on smithi118 to 3946M 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:07.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:07.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:07 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67228]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:20:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67228]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67228]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67228]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:07.828 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67248]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:20:07.828 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67248]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:07.828 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67248]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:07.828 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67248]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:08.088 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67262]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:20:08.088 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67262]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:08.088 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67262]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:08.088 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:20:07 smithi118 sudo[67262]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:08.315 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:08.312+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7b480564f0 con 0x7f7b5c101cb0 2024-03-12T03:20:08.350 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67354]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:20:08.350 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67354]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:08.350 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67354]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:08.350 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67354]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:08.350 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:08.350 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 ceph-mon[29160]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T03:20:08.350 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1173909933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d46b2ef-fba5-4437-b107-9a321bb94874"}]: dispatch 2024-03-12T03:20:08.350 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 ceph-mon[29160]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d46b2ef-fba5-4437-b107-9a321bb94874"}]: dispatch 2024-03-12T03:20:08.351 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 ceph-mon[29160]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d46b2ef-fba5-4437-b107-9a321bb94874"}]': finished 2024-03-12T03:20:08.351 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 ceph-mon[29160]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:20:08.351 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:08.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:08 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:08.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:08 smithi177 ceph-mon[37140]: osdmap e23: 3 total, 3 up, 3 in 2024-03-12T03:20:08.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:08 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/1173909933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d46b2ef-fba5-4437-b107-9a321bb94874"}]: dispatch 2024-03-12T03:20:08.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:08 smithi177 ceph-mon[37140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d46b2ef-fba5-4437-b107-9a321bb94874"}]: dispatch 2024-03-12T03:20:08.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:08 smithi177 ceph-mon[37140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d46b2ef-fba5-4437-b107-9a321bb94874"}]': finished 2024-03-12T03:20:08.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:08 smithi177 ceph-mon[37140]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:20:08.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:08 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:08.601 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67378]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:20:08.602 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67378]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:08.602 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67378]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:08.602 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67378]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:08.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67383]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:20:08.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67383]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:08.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67383]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:08.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67383]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:09.121 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67433]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:20:09.121 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67433]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:09.121 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67433]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:09.121 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:08 smithi118 sudo[67433]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:09.376 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41092]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:20:09.376 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41092]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:09.376 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41092]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:09.376 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41092]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:09.384 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: mgrmap e19: smithi118.zzzuvl(active, since 2m), standbys: smithi177.tmmsuz 2024-03-12T03:20:09.385 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:20:09.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41283]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41283]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41283]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 sudo[41283]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: mgrmap e19: smithi118.zzzuvl(active, since 2m), standbys: smithi177.tmmsuz 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/1170497436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:20:09.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:20:09.776 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1170497436' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:20:09.777 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:09.777 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:20:09.777 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:20:09.777 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:20:09.777 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:20:09.777 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:20:10.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:10 smithi177 ceph-mon[37140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:20:10.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:10 smithi177 ceph-mon[37140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:20:10.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:10 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:10.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:10 smithi118 ceph-mon[29160]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:20:10.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:10 smithi118 ceph-mon[29160]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:20:10.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:10 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:11.662 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:11 smithi177 ceph-mon[37140]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:11.663 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:11.663 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:11 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:11 smithi118 ceph-mon[29160]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:11 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:12.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:12 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:12 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:12 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:12.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:12 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:12.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:12 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:12.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:12 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:13.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:13 smithi177 ceph-mon[37140]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:13.712 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:13 smithi118 ceph-mon[29160]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:15.164 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:14 smithi177 ceph-mon[37140]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:15.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:14 smithi118 ceph-mon[29160]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:16.009 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:15 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:20:16.009 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:15 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:16.010 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:15 smithi177 ceph-mon[37140]: Deploying daemon osd.3 on smithi177 2024-03-12T03:20:16.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:15 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:20:16.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:15 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:16.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:15 smithi118 ceph-mon[29160]: Deploying daemon osd.3 on smithi177 2024-03-12T03:20:16.949 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:16 smithi177 ceph-mon[37140]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:17.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:16 smithi118 ceph-mon[29160]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:18.877 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:18.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:19.915 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:19 smithi177 ceph-mon[37140]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:20.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:19 smithi118 ceph-mon[29160]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:21.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:21 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:21.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:21 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:21.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:21 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:21.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:21 smithi118 ceph-mon[29160]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:21.872 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:21 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:21.872 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:21 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:21.872 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:21 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:21.872 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:21 smithi177 ceph-mon[37140]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:23.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:23 smithi177 ceph-mon[37140]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:24.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:23 smithi118 ceph-mon[29160]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:24.325 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.322+0000 7f7b59ffb700 1 -- 172.21.15.177:0/451455746 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7b5c0611f0 con 0x7f7b4c06c800 2024-03-12T03:20:24.325 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 3 on host 'smithi177' 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7b4c06c800 msgr2=0x7f7b4c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7b4c06c800 0x7f7b4c06ecc0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7b50000c00 tx=0x7f7b50005960 comp rx=0 tx=0).stop 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c101cb0 msgr2=0x7f7b5c196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c101cb0 0x7f7b5c196a50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7b4800d8d0 tx=0x7f7b4800dbe0 comp rx=0 tx=0).stop 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 shutdown_connections 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7b4c06c800 0x7f7b4c06ecc0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b5c100ab0 0x7f7b5c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 --2- 172.21.15.177:0/451455746 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7b5c101cb0 0x7f7b5c196a50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:24.327 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.324+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 >> 172.21.15.177:0/451455746 conn(0x7f7b5c0fc090 msgr2=0x7f7b5c1052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:20:24.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.325+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 shutdown_connections 2024-03-12T03:20:24.328 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:24.325+0000 7f7b62cd6700 1 -- 172.21.15.177:0/451455746 wait complete. 2024-03-12T03:20:24.983 DEBUG:teuthology.orchestra.run.smithi177:osd.3> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T03:20:24.986 INFO:tasks.cephadm:Deploying osd.4 on smithi177 with /dev/vg_nvme/lv_3... 2024-03-12T03:20:24.986 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:20:25.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:25 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:25 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:25 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:25 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:25 smithi118 ceph-mon[29160]: from='osd.3 [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:20:25.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:25 smithi118 ceph-mon[29160]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:20:25.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:25 smithi118 ceph-mon[29160]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:25.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:25 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:25 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:25 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:25 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:25.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:25 smithi177 ceph-mon[37140]: from='osd.3 [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:20:25.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:25 smithi177 ceph-mon[37140]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:20:25.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:25 smithi177 ceph-mon[37140]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:26.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:26 smithi177 ceph-mon[37140]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:20:26.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:26 smithi177 ceph-mon[37140]: osdmap e25: 4 total, 3 up, 4 in 2024-03-12T03:20:26.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:26 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:26.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:26 smithi177 ceph-mon[37140]: from='osd.3 [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:26.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:26 smithi177 ceph-mon[37140]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:26.453 INFO:journalctl@ceph.osd.3.smithi177.stdout:Mar 12 03:20:26 smithi177 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-3[43134]: 2024-03-12T03:20:26.372+0000 7f39a659a700 -1 osd.3 0 waiting for initial osdmap 2024-03-12T03:20:26.453 INFO:journalctl@ceph.osd.3.smithi177.stdout:Mar 12 03:20:26 smithi177 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-3[43134]: 2024-03-12T03:20:26.378+0000 7f39a038b700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:20:26.626 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi177/config 2024-03-12T03:20:26.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:26 smithi118 ceph-mon[29160]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:20:26.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:26 smithi118 ceph-mon[29160]: osdmap e25: 4 total, 3 up, 4 in 2024-03-12T03:20:26.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:26 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:26.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:26 smithi118 ceph-mon[29160]: from='osd.3 [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:26.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:26 smithi118 ceph-mon[29160]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:27.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:27 smithi177 ceph-mon[37140]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-12T03:20:27.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:27 smithi177 ceph-mon[37140]: osdmap e26: 4 total, 3 up, 4 in 2024-03-12T03:20:27.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:27.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:27 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:27.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:27 smithi177 ceph-mon[37140]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:27 smithi118 ceph-mon[29160]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-12T03:20:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:27 smithi118 ceph-mon[29160]: osdmap e26: 4 total, 3 up, 4 in 2024-03-12T03:20:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:27 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:27.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:27 smithi118 ceph-mon[29160]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:20:28.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: purged_snaps scrub starts 2024-03-12T03:20:28.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: purged_snaps scrub ok 2024-03-12T03:20:28.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: osd.3 [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] boot 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:28.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:28 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: purged_snaps scrub starts 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: purged_snaps scrub ok 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: osd.3 [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] boot 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: osdmap e27: 4 total, 4 up, 4 in 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:28.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:28.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:28.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:28.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:28 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:29.041 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:20:29.041 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:20:29.041 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-12T03:20:29.041 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-12T03:20:29.041 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012267 s, 855 MB/s 2024-03-12T03:20:29.041 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-12T03:20:29.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: Detected new or changed devices on smithi177 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: Adjusting osd_memory_target on smithi177 to 14912M 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:29.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:29 smithi177 ceph-mon[37140]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T03:20:29.728 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: Detected new or changed devices on smithi177 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: Adjusting osd_memory_target on smithi177 to 14912M 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: osdmap e28: 4 total, 4 up, 4 in 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:29 smithi118 ceph-mon[29160]: osdmap e29: 4 total, 4 up, 4 in 2024-03-12T03:20:29.931 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi177/config 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 -- 172.21.15.177:0/1007503485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024101cf0 msgr2=0x7f1024102150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 --2- 172.21.15.177:0/1007503485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024101cf0 0x7f1024102150 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1014009910 tx=0x7f1014009c20 comp rx=0 tx=0).stop 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 -- 172.21.15.177:0/1007503485 shutdown_connections 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 --2- 172.21.15.177:0/1007503485 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024101cf0 0x7f1024102150 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 --2- 172.21.15.177:0/1007503485 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1024100af0 0x7f1024100f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 -- 172.21.15.177:0/1007503485 >> 172.21.15.177:0/1007503485 conn(0x7f10240fc0b0 msgr2=0x7f10240fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 -- 172.21.15.177:0/1007503485 shutdown_connections 2024-03-12T03:20:31.440 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.435+0000 7f102b584700 1 -- 172.21.15.177:0/1007503485 wait complete. 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 Processor -- start 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 -- start start 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024100af0 0x7f10241964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1024101cf0 0x7f10241969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1024197000 con 0x7f1024101cf0 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1024197140 con 0x7f1024100af0 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1029320700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024100af0 0x7f10241964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1029320700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024100af0 0x7f10241964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:56234/0 (socket says 172.21.15.177:56234) 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1029320700 1 -- 172.21.15.177:0/1900164799 learned_addr learned my addr 172.21.15.177:0/1900164799 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1029320700 1 -- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1024101cf0 msgr2=0x7f10241969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1028b1f700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1024101cf0 0x7f10241969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1029320700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1024101cf0 0x7f10241969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1029320700 1 -- 172.21.15.177:0/1900164799 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10140095f0 con 0x7f1024100af0 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f1029320700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024100af0 0x7f10241964a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f102000d900 tx=0x7f102000dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:20:31.441 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f101a7fc700 1 -- 172.21.15.177:0/1900164799 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102000b920 con 0x7f1024100af0 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f101a7fc700 1 -- 172.21.15.177:0/1900164799 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1020010460 con 0x7f1024100af0 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102419bba0 con 0x7f1024100af0 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f101a7fc700 1 -- 172.21.15.177:0/1900164799 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f102000f590 con 0x7f1024100af0 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.436+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102419c0f0 con 0x7f1024100af0 2024-03-12T03:20:31.442 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.437+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1024190670 con 0x7f1024100af0 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.438+0000 7f101a7fc700 1 -- 172.21.15.177:0/1900164799 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f10200105d0 con 0x7f1024100af0 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.438+0000 7f101a7fc700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f101006c880 0x7f101006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.438+0000 7f101a7fc700 1 -- 172.21.15.177:0/1900164799 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3774+0+0 (secure 0 0 0) 0x7f102008b540 con 0x7f1024100af0 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.438+0000 7f1028b1f700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f101006c880 0x7f101006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:31.443 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.439+0000 7f1028b1f700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f101006c880 0x7f101006ed40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f101400b5c0 tx=0x7f1014005160 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:20:31.445 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.443+0000 7f101a7fc700 1 -- 172.21.15.177:0/1900164799 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1020059c80 con 0x7f1024100af0 2024-03-12T03:20:31.646 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:31.643+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f10240611f0 con 0x7f101006c880 2024-03-12T03:20:31.646 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:31 smithi177 ceph-mon[37140]: osdmap e30: 4 total, 4 up, 4 in 2024-03-12T03:20:31.646 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:31 smithi177 ceph-mon[37140]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:20:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:31 smithi118 ceph-mon[29160]: osdmap e30: 4 total, 4 up, 4 in 2024-03-12T03:20:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:31 smithi118 ceph-mon[29160]: pgmap v99: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-12T03:20:32.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:32 smithi177 ceph-mon[37140]: osdmap e31: 4 total, 4 up, 4 in 2024-03-12T03:20:32.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:32 smithi177 ceph-mon[37140]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:32.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:32 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:32.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:32 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:32.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:32 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:32.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:32 smithi118 ceph-mon[29160]: osdmap e31: 4 total, 4 up, 4 in 2024-03-12T03:20:32.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:32 smithi118 ceph-mon[29160]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:32.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:32 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:32.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:32 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:32.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:32 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:33.477 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:33 smithi177 ceph-mon[37140]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-12T03:20:33.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:33 smithi118 ceph-mon[29160]: pgmap v101: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-12T03:20:34.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:34 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:34.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:34 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/3858257315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b56220a9-5e7e-443a-b2c2-abcbca3b0c8d"}]: dispatch 2024-03-12T03:20:34.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:34 smithi177 ceph-mon[37140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b56220a9-5e7e-443a-b2c2-abcbca3b0c8d"}]: dispatch 2024-03-12T03:20:34.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:34 smithi177 ceph-mon[37140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b56220a9-5e7e-443a-b2c2-abcbca3b0c8d"}]': finished 2024-03-12T03:20:34.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:34 smithi177 ceph-mon[37140]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T03:20:34.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:34 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:34 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:34 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/3858257315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b56220a9-5e7e-443a-b2c2-abcbca3b0c8d"}]: dispatch 2024-03-12T03:20:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:34 smithi118 ceph-mon[29160]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b56220a9-5e7e-443a-b2c2-abcbca3b0c8d"}]: dispatch 2024-03-12T03:20:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:34 smithi118 ceph-mon[29160]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b56220a9-5e7e-443a-b2c2-abcbca3b0c8d"}]': finished 2024-03-12T03:20:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:34 smithi118 ceph-mon[29160]: osdmap e32: 5 total, 4 up, 5 in 2024-03-12T03:20:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:34 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:35.688 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:35 smithi177 ceph-mon[37140]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-12T03:20:35.688 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:35 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/2039236888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:20:35.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:35 smithi118 ceph-mon[29160]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-12T03:20:35.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:35 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/2039236888' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:20:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:37 smithi177 ceph-mon[37140]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:38.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:37 smithi118 ceph-mon[29160]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:39.838 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:39 smithi177 ceph-mon[37140]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:40.015 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:39 smithi118 ceph-mon[29160]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:41.871 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:41 smithi177 ceph-mon[37140]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:41.871 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:41 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:20:41.872 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:41 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:42.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:41 smithi118 ceph-mon[29160]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:42.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:41 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:20:42.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:41 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:42.785 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:42 smithi177 ceph-mon[37140]: Deploying daemon osd.4 on smithi177 2024-03-12T03:20:43.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:42 smithi118 ceph-mon[29160]: Deploying daemon osd.4 on smithi177 2024-03-12T03:20:43.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:43 smithi177 ceph-mon[37140]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:43.961 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:43 smithi118 ceph-mon[29160]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:45.139 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:44 smithi177 ceph-mon[37140]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:45.232 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:44 smithi118 ceph-mon[29160]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:47.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:47 smithi177 ceph-mon[37140]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:47.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:47 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:47.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:47 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:47.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:47 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:48.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:47 smithi118 ceph-mon[29160]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:48.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:47 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:48.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:47 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:48.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:47 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:48.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:48.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:20:49.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:49 smithi177 ceph-mon[37140]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:50.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:49 smithi118 ceph-mon[29160]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:50.508 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.505+0000 7f101a7fc700 1 -- 172.21.15.177:0/1900164799 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f10240611f0 con 0x7f101006c880 2024-03-12T03:20:50.508 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 4 on host 'smithi177' 2024-03-12T03:20:50.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f101006c880 msgr2=0x7f101006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:50.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f101006c880 0x7f101006ed40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f101400b5c0 tx=0x7f1014005160 comp rx=0 tx=0).stop 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024100af0 msgr2=0x7f10241964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024100af0 0x7f10241964a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f102000d900 tx=0x7f102000dcc0 comp rx=0 tx=0).stop 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 shutdown_connections 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f101006c880 0x7f101006ed40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1024100af0 0x7f10241964a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 --2- 172.21.15.177:0/1900164799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1024101cf0 0x7f10241969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 >> 172.21.15.177:0/1900164799 conn(0x7f10240fc0b0 msgr2=0x7f10241052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 shutdown_connections 2024-03-12T03:20:50.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:50.508+0000 7f102b584700 1 -- 172.21.15.177:0/1900164799 wait complete. 2024-03-12T03:20:51.350 DEBUG:teuthology.orchestra.run.smithi177:osd.4> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.4.service 2024-03-12T03:20:51.353 INFO:tasks.cephadm:Deploying osd.5 on smithi177 with /dev/vg_nvme/lv_2... 2024-03-12T03:20:51.353 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:20:51.603 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi177/config 2024-03-12T03:20:51.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:51 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:51.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:51 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:51.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:51 smithi177 ceph-mon[37140]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:51.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:51 smithi177 ceph-mon[37140]: from='osd.4 [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:20:51.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:51 smithi177 ceph-mon[37140]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:20:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:51 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:51 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:51 smithi118 ceph-mon[29160]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:51 smithi118 ceph-mon[29160]: from='osd.4 [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:20:51.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:51 smithi118 ceph-mon[29160]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:20:52.804 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-mon[37140]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:20:52.804 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-mon[37140]: osdmap e33: 5 total, 4 up, 5 in 2024-03-12T03:20:52.804 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:52.804 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-mon[37140]: from='osd.4 [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:52.804 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-mon[37140]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:52.804 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:52.804 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:52.805 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-4[50076]: 2024-03-12T03:20:52.509+0000 7f86cb6a2700 -1 osd.4 0 waiting for initial osdmap 2024-03-12T03:20:52.805 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 12 03:20:52 smithi177 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-4[50076]: 2024-03-12T03:20:52.516+0000 7f86c5493700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:20:52.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:52 smithi118 ceph-mon[29160]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:20:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:52 smithi118 ceph-mon[29160]: osdmap e33: 5 total, 4 up, 5 in 2024-03-12T03:20:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:52 smithi118 ceph-mon[29160]: from='osd.4 [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:52 smithi118 ceph-mon[29160]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:20:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:53.650 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:53 smithi177 ceph-mon[37140]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-12T03:20:53.650 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:53 smithi177 ceph-mon[37140]: osdmap e34: 5 total, 4 up, 5 in 2024-03-12T03:20:53.650 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:53.650 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:53.650 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:53 smithi177 ceph-mon[37140]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:53.650 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:20:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:20:53.651 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-12T03:20:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-12T03:20:53.651 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123479 s, 849 MB/s 2024-03-12T03:20:53.651 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-12T03:20:53.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:53 smithi118 ceph-mon[29160]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-12T03:20:53.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:53 smithi118 ceph-mon[29160]: osdmap e34: 5 total, 4 up, 5 in 2024-03-12T03:20:53.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:53.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:53.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:53 smithi118 ceph-mon[29160]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:20:54.556 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-03-12T03:20:54.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:54 smithi177 ceph-mon[37140]: purged_snaps scrub starts 2024-03-12T03:20:54.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:54 smithi177 ceph-mon[37140]: purged_snaps scrub ok 2024-03-12T03:20:54.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:54 smithi177 ceph-mon[37140]: osd.4 [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] boot 2024-03-12T03:20:54.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:54 smithi177 ceph-mon[37140]: osdmap e35: 5 total, 5 up, 5 in 2024-03-12T03:20:54.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:54.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:54 smithi118 ceph-mon[29160]: purged_snaps scrub starts 2024-03-12T03:20:54.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:54 smithi118 ceph-mon[29160]: purged_snaps scrub ok 2024-03-12T03:20:54.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:54 smithi118 ceph-mon[29160]: osd.4 [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] boot 2024-03-12T03:20:54.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:54 smithi118 ceph-mon[29160]: osdmap e35: 5 total, 5 up, 5 in 2024-03-12T03:20:54.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:20:55.443 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi177/config 2024-03-12T03:20:55.562 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:55 smithi177 ceph-mon[37140]: osdmap e36: 5 total, 5 up, 5 in 2024-03-12T03:20:55.563 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:55 smithi177 ceph-mon[37140]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:20:55.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:55 smithi118 ceph-mon[29160]: osdmap e36: 5 total, 5 up, 5 in 2024-03-12T03:20:55.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:55 smithi118 ceph-mon[29160]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:20:57.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.636+0000 7f146cb56700 1 -- 172.21.15.177:0/663307790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 msgr2=0x7f1468100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:57.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.636+0000 7f146cb56700 1 --2- 172.21.15.177:0/663307790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f1468100fe0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f145c009910 tx=0x7f145c009c20 comp rx=0 tx=0).stop 2024-03-12T03:20:57.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 -- 172.21.15.177:0/663307790 shutdown_connections 2024-03-12T03:20:57.640 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 --2- 172.21.15.177:0/663307790 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f1468100fe0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 --2- 172.21.15.177:0/663307790 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14680ff980 0x7f14680ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 -- 172.21.15.177:0/663307790 >> 172.21.15.177:0/663307790 conn(0x7f14680faf60 msgr2=0x7f14680fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 -- 172.21.15.177:0/663307790 shutdown_connections 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 -- 172.21.15.177:0/663307790 wait complete. 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 Processor -- start 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 -- start start 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14680ff980 0x7f1468068f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f146819b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146819b720 con 0x7f14680ff980 2024-03-12T03:20:57.641 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.637+0000 7f146cb56700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146819b890 con 0x7f1468100b80 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f1465d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f146819b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f146659c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14680ff980 0x7f1468068f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f1465d9b700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f146819b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:46182/0 (socket says 172.21.15.177:46182) 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f1465d9b700 1 -- 172.21.15.177:0/3338384130 learned_addr learned my addr 172.21.15.177:0/3338384130 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f1465d9b700 1 -- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14680ff980 msgr2=0x7f1468068f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f1465d9b700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14680ff980 0x7f1468068f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f1465d9b700 1 -- 172.21.15.177:0/3338384130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145c0095f0 con 0x7f1468100b80 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f1465d9b700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f146819b1e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f145c0092e0 tx=0x7f145c00bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f14577fe700 1 -- 172.21.15.177:0/3338384130 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f145c01d070 con 0x7f1468100b80 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146819bb10 con 0x7f1468100b80 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f14577fe700 1 -- 172.21.15.177:0/3338384130 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f145c00f460 con 0x7f1468100b80 2024-03-12T03:20:57.642 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f14577fe700 1 -- 172.21.15.177:0/3338384130 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f145c021620 con 0x7f1468100b80 2024-03-12T03:20:57.643 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.638+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146819c080 con 0x7f1468100b80 2024-03-12T03:20:57.643 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.639+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f146818f500 con 0x7f1468100b80 2024-03-12T03:20:57.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.640+0000 7f14577fe700 1 -- 172.21.15.177:0/3338384130 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f145c00f5d0 con 0x7f1468100b80 2024-03-12T03:20:57.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.640+0000 7f14577fe700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f145006c880 0x7f145006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:20:57.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.640+0000 7f14577fe700 1 -- 172.21.15.177:0/3338384130 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f145c08c9a0 con 0x7f1468100b80 2024-03-12T03:20:57.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.640+0000 7f146659c700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f145006c880 0x7f145006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:20:57.644 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.641+0000 7f146659c700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f145006c880 0x7f145006ed40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f14681009e0 tx=0x7f145800b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:20:57.646 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.645+0000 7f14577fe700 1 -- 172.21.15.177:0/3338384130 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f145c05af30 con 0x7f1468100b80 2024-03-12T03:20:57.848 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:20:57.846+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f14680611f0 con 0x7f145006c880 2024-03-12T03:20:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:20:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:58.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:20:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:20:58.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:20:58.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:58 smithi177 ceph-mon[37140]: Detected new or changed devices on smithi177 2024-03-12T03:20:58.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:58 smithi177 ceph-mon[37140]: Adjusting osd_memory_target on smithi177 to 7456M 2024-03-12T03:20:58.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:58.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:58.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:58 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:58.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:58 smithi118 ceph-mon[29160]: Detected new or changed devices on smithi177 2024-03-12T03:20:58.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:58 smithi118 ceph-mon[29160]: Adjusting osd_memory_target on smithi177 to 7456M 2024-03-12T03:20:58.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:20:58.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:20:58.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:58 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:20:59.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:59 smithi177 ceph-mon[37140]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:20:59.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:20:59 smithi177 ceph-mon[37140]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:00.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:59 smithi118 ceph-mon[29160]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:21:00.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:20:59 smithi118 ceph-mon[29160]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:00.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:00 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/1132997625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe10ad0-409a-4af7-84da-9f6eee5214bc"}]: dispatch 2024-03-12T03:21:00.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:00 smithi177 ceph-mon[37140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe10ad0-409a-4af7-84da-9f6eee5214bc"}]: dispatch 2024-03-12T03:21:00.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:00 smithi177 ceph-mon[37140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe10ad0-409a-4af7-84da-9f6eee5214bc"}]': finished 2024-03-12T03:21:00.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:00 smithi177 ceph-mon[37140]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:21:00.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:00 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:01.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:00 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1132997625' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe10ad0-409a-4af7-84da-9f6eee5214bc"}]: dispatch 2024-03-12T03:21:01.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:00 smithi118 ceph-mon[29160]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe10ad0-409a-4af7-84da-9f6eee5214bc"}]: dispatch 2024-03-12T03:21:01.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:00 smithi118 ceph-mon[29160]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe10ad0-409a-4af7-84da-9f6eee5214bc"}]': finished 2024-03-12T03:21:01.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:00 smithi118 ceph-mon[29160]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:21:01.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:00 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:01.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:01 smithi177 ceph-mon[37140]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:01.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:01 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/1843782624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:02.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:01 smithi118 ceph-mon[29160]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:02.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:01 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1843782624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:21:03.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:03 smithi177 ceph-mon[37140]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:03.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:04.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:03 smithi118 ceph-mon[29160]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:04.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:05.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:04 smithi177 ceph-mon[37140]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:05.204 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:04 smithi118 ceph-mon[29160]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:07.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:07 smithi177 ceph-mon[37140]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:08.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:07 smithi118 ceph-mon[29160]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:08.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:08 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:21:08.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:08 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:08.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:08 smithi177 ceph-mon[37140]: Deploying daemon osd.5 on smithi177 2024-03-12T03:21:08.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:08 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:21:08.976 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:08 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:08.976 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:08 smithi118 ceph-mon[29160]: Deploying daemon osd.5 on smithi177 2024-03-12T03:21:09.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:09 smithi177 ceph-mon[37140]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:10.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:09 smithi118 ceph-mon[29160]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:11.951 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:11 smithi177 ceph-mon[37140]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:12.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:11 smithi118 ceph-mon[29160]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:13.756 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:13 smithi177 ceph-mon[37140]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:13.756 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:13 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:13.756 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:13 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:13.756 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:13 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:13 smithi118 ceph-mon[29160]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:13 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:13 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:13 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:15.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:14 smithi177 ceph-mon[37140]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:15.281 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:14 smithi118 ceph-mon[29160]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:17.116 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.114+0000 7f14577fe700 1 -- 172.21.15.177:0/3338384130 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f14680611f0 con 0x7f145006c880 2024-03-12T03:21:17.116 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 5 on host 'smithi177' 2024-03-12T03:21:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f145006c880 msgr2=0x7f145006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f145006c880 0x7f145006ed40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f14681009e0 tx=0x7f145800b410 comp rx=0 tx=0).stop 2024-03-12T03:21:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 msgr2=0x7f146819b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:17.119 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f146819b1e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f145c0092e0 tx=0x7f145c00bd70 comp rx=0 tx=0).stop 2024-03-12T03:21:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 shutdown_connections 2024-03-12T03:21:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f145006c880 0x7f145006ed40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f14680ff980 0x7f1468068f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 --2- 172.21.15.177:0/3338384130 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1468100b80 0x7f146819b1e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 >> 172.21.15.177:0/3338384130 conn(0x7f14680faf60 msgr2=0x7f1468104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 shutdown_connections 2024-03-12T03:21:17.120 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:17.117+0000 7f146cb56700 1 -- 172.21.15.177:0/3338384130 wait complete. 2024-03-12T03:21:17.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:17 smithi177 ceph-mon[37140]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:17.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:17 smithi177 ceph-mon[37140]: from='osd.5 [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:21:17.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:17 smithi177 ceph-mon[37140]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:21:17.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:17 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:17.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:17 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:17.798 DEBUG:teuthology.orchestra.run.smithi177:osd.5> sudo journalctl -f -n 0 -u ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.5.service 2024-03-12T03:21:17.801 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-12T03:21:17.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T03:21:18.025 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:18.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:17 smithi118 ceph-mon[29160]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:18.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:17 smithi118 ceph-mon[29160]: from='osd.5 [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:21:18.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:17 smithi118 ceph-mon[29160]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:21:18.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:17 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:18.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:17 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:18.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:21:18.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: osdmap e38: 6 total, 5 up, 6 in 2024-03-12T03:21:18.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:18.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: from='osd.5 [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:21:18.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:21:18.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:18.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:18.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:18.886 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:21:18.886 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: osdmap e38: 6 total, 5 up, 6 in 2024-03-12T03:21:18.886 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:18.886 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: from='osd.5 [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:21:18.886 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-12T03:21:18.887 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:18.887 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:18.887 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:18.887 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-5[56612]: 2024-03-12T03:21:18.618+0000 7f60daf61700 -1 osd.5 0 waiting for initial osdmap 2024-03-12T03:21:18.887 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 12 03:21:18 smithi177 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-5[56612]: 2024-03-12T03:21:18.626+0000 7f60d7557700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:21:19.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 -- 172.21.15.118:0/1309691316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 msgr2=0x7f604c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:19.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 --2- 172.21.15.118:0/1309691316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c1024c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f603c009910 tx=0x7f603c009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:19.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 -- 172.21.15.118:0/1309691316 shutdown_connections 2024-03-12T03:21:19.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 --2- 172.21.15.118:0/1309691316 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f604c0feb20 0x7f604c0fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:19.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 --2- 172.21.15.118:0/1309691316 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c1024c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:19.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 -- 172.21.15.118:0/1309691316 >> 172.21.15.118:0/1309691316 conn(0x7f604c0fa6e0 msgr2=0x7f604c0fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:19.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 -- 172.21.15.118:0/1309691316 shutdown_connections 2024-03-12T03:21:19.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 -- 172.21.15.118:0/1309691316 wait complete. 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 Processor -- start 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.719+0000 7f6050d85700 1 -- start start 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6050d85700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f604c0feb20 0x7f604c101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6050d85700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c0ffdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6050d85700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c100330 con 0x7f604c1020a0 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6050d85700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604c100470 con 0x7f604c0feb20 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6049d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c0ffdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f604a59c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f604c0feb20 0x7f604c101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6049d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c0ffdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44618/0 (socket says 172.21.15.118:44618) 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6049d9b700 1 -- 172.21.15.118:0/3347191883 learned_addr learned my addr 172.21.15.118:0/3347191883 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f604a59c700 1 -- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 msgr2=0x7f604c0ffdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f604a59c700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c0ffdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:19.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f604a59c700 1 -- 172.21.15.118:0/3347191883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c0095f0 con 0x7f604c0feb20 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6049d9b700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c0ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f604a59c700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f604c0feb20 0x7f604c101750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f603c006010 tx=0x7f603c005e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f603b7fe700 1 -- 172.21.15.118:0/3347191883 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f603c01d070 con 0x7f604c0feb20 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f603b7fe700 1 -- 172.21.15.118:0/3347191883 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f603c0052a0 con 0x7f604c0feb20 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604c1006f0 con 0x7f604c0feb20 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f603b7fe700 1 -- 172.21.15.118:0/3347191883 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f603c003e70 con 0x7f604c0feb20 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.720+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f604c100c60 con 0x7f604c0feb20 2024-03-12T03:21:19.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.721+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f604c190650 con 0x7f604c0feb20 2024-03-12T03:21:19.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.722+0000 7f603b7fe700 1 -- 172.21.15.118:0/3347191883 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f603c003710 con 0x7f604c0feb20 2024-03-12T03:21:19.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.722+0000 7f603b7fe700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f603406c820 0x7f603406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:19.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.722+0000 7f6049d9b700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f603406c820 0x7f603406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:19.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.722+0000 7f603b7fe700 1 -- 172.21.15.118:0/3347191883 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f603c08cae0 con 0x7f604c0feb20 2024-03-12T03:21:19.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.723+0000 7f6049d9b700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f603406c820 0x7f603406ece0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6040011550 tx=0x7f6040009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:19.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.727+0000 7f603b7fe700 1 -- 172.21.15.118:0/3347191883 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f603c05ae20 con 0x7f604c0feb20 2024-03-12T03:21:19.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.919+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f604c100c60 con 0x7f604c0feb20 2024-03-12T03:21:19.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:19 smithi118 ceph-mon[29160]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:19.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:19 smithi118 ceph-mon[29160]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-12T03:21:19.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:19 smithi118 ceph-mon[29160]: osdmap e39: 6 total, 5 up, 6 in 2024-03-12T03:21:19.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:19.923 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:19 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:19.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.920+0000 7f603b7fe700 1 -- 172.21.15.118:0/3347191883 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f604c100c60 con 0x7f604c0feb20 2024-03-12T03:21:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f603406c820 msgr2=0x7f603406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f603406c820 0x7f603406ece0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6040011550 tx=0x7f6040009260 comp rx=0 tx=0).stop 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f604c0feb20 msgr2=0x7f604c101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f604c0feb20 0x7f604c101750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f603c006010 tx=0x7f603c005e30 comp rx=0 tx=0).stop 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 shutdown_connections 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f603406c820 0x7f603406ece0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f604c0feb20 0x7f604c101750 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 --2- 172.21.15.118:0/3347191883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f604c1020a0 0x7f604c0ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 >> 172.21.15.118:0/3347191883 conn(0x7f604c0fa6e0 msgr2=0x7f604c1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:19.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 shutdown_connections 2024-03-12T03:21:19.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:19.922+0000 7f6050d85700 1 -- 172.21.15.118:0/3347191883 wait complete. 2024-03-12T03:21:19.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:19 smithi177 ceph-mon[37140]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:19.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:19 smithi177 ceph-mon[37140]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-12T03:21:19.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:19 smithi177 ceph-mon[37140]: osdmap e39: 6 total, 5 up, 6 in 2024-03-12T03:21:19.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:19.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:19 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:20.612 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1710213679,"num_in_osds":6,"osd_in_since":1710213660,"num_remapped_pgs":0} 2024-03-12T03:21:20.613 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:21:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:20.885 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:20 smithi118 ceph-mon[29160]: purged_snaps scrub starts 2024-03-12T03:21:20.885 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:20 smithi118 ceph-mon[29160]: purged_snaps scrub ok 2024-03-12T03:21:20.885 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:20 smithi118 ceph-mon[29160]: osd.5 [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] boot 2024-03-12T03:21:20.886 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:20 smithi118 ceph-mon[29160]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:21:20.886 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:20 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:20.886 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:20 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3347191883' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:20.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:20 smithi177 ceph-mon[37140]: purged_snaps scrub starts 2024-03-12T03:21:20.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:20 smithi177 ceph-mon[37140]: purged_snaps scrub ok 2024-03-12T03:21:20.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:20 smithi177 ceph-mon[37140]: osd.5 [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] boot 2024-03-12T03:21:20.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:20 smithi177 ceph-mon[37140]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:21:20.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:20 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:21:20.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:20 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3347191883' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:21:21.677 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:21 smithi177 ceph-mon[37140]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:21.678 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:21 smithi177 ceph-mon[37140]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:21:21.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:21 smithi118 ceph-mon[29160]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:21:21.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:21 smithi118 ceph-mon[29160]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:21:22.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/1894665983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a180ff940 msgr2=0x7f8a180ffdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:22.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/1894665983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a180ff940 0x7f8a180ffdc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8a08009910 tx=0x7f8a08009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/1894665983 shutdown_connections 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/1894665983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a180ff940 0x7f8a180ffdc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/1894665983 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a18105940 0x7f8a18105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/1894665983 >> 172.21.15.118:0/1894665983 conn(0x7f8a180fb450 msgr2=0x7f8a180fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/1894665983 shutdown_connections 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.440+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/1894665983 wait complete. 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1fc7c700 1 Processor -- start 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1fc7c700 1 -- start start 2024-03-12T03:21:22.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1fc7c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a180ff940 0x7f8a18195610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1fc7c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a18105940 0x7f8a18195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1fc7c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18069e50 con 0x7f8a18105940 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1fc7c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a18196090 con 0x7f8a180ff940 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1d217700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a18105940 0x7f8a18195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1d217700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a18105940 0x7f8a18195b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44644/0 (socket says 172.21.15.118:44644) 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.441+0000 7f8a1d217700 1 -- 172.21.15.118:0/2582573535 learned_addr learned my addr 172.21.15.118:0/2582573535 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a1da18700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a180ff940 0x7f8a18195610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a1d217700 1 -- 172.21.15.118:0/2582573535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a180ff940 msgr2=0x7f8a18195610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a1d217700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a180ff940 0x7f8a18195610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a1d217700 1 -- 172.21.15.118:0/2582573535 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a080095f0 con 0x7f8a18105940 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a1d217700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a18105940 0x7f8a18195b50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800bd70 tx=0x7f8a0800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a0effd700 1 -- 172.21.15.118:0/2582573535 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a0801d070 con 0x7f8a18105940 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a18196310 con 0x7f8a18105940 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a0effd700 1 -- 172.21.15.118:0/2582573535 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a080056e0 con 0x7f8a18105940 2024-03-12T03:21:22.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a0effd700 1 -- 172.21.15.118:0/2582573535 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a08017400 con 0x7f8a18105940 2024-03-12T03:21:22.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.442+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a181a2e50 con 0x7f8a18105940 2024-03-12T03:21:22.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.443+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a18107f60 con 0x7f8a18105940 2024-03-12T03:21:22.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.443+0000 7f8a0effd700 1 -- 172.21.15.118:0/2582573535 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8a08003a60 con 0x7f8a18105940 2024-03-12T03:21:22.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.444+0000 7f8a0effd700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f8a0406c880 0x7f8a0406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:22.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.444+0000 7f8a0effd700 1 -- 172.21.15.118:0/2582573535 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f8a0808cda0 con 0x7f8a18105940 2024-03-12T03:21:22.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.444+0000 7f8a1da18700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f8a0406c880 0x7f8a0406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:22.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.444+0000 7f8a1da18700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f8a0406c880 0x7f8a0406ed40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009d10 tx=0x7f8a10009040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:22.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.449+0000 7f8a0effd700 1 -- 172.21.15.118:0/2582573535 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8a080576f0 con 0x7f8a18105940 2024-03-12T03:21:22.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.648+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a18101080 con 0x7f8a18105940 2024-03-12T03:21:22.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.648+0000 7f8a0effd700 1 -- 172.21.15.118:0/2582573535 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f8a0805ad10 con 0x7f8a18105940 2024-03-12T03:21:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:21:22.652 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":41,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","created":"2024-03-12T03:15:23.858621+0000","modified":"2024-03-12T03:21:20.621261+0000","last_up_change":"2024-03-12T03:21:19.616038+0000","last_in_change":"2024-03-12T03:21:00.310107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:20:04.838424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96b74c65-8c14-4178-9e8d-b981032baa7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6803","nonce":1275003428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6805","nonce":1275003428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6809","nonce":1275003428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6807","nonce":1275003428}]},"public_addr":"172.21.15.118:6803/1275003428","cluster_addr":"172.21.15.118:6805/1275003428","heartbeat_back_addr":"172.21.15.118:6809/1275003428","heartbeat_front_addr":"172.21.15.118:6807/1275003428","state":["exists","up"]},{"osd":1,"uuid":"3c911034-856e-43e7-b267-2a6c63cf46fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6811","nonce":1159607125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6813","nonce":1159607125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6817","nonce":1159607125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6815","nonce":1159607125}]},"public_addr":"172.21.15.118:6811/1159607125","cluster_addr":"172.21.15.118:6813/1159607125","heartbeat_back_addr":"172.21.15.118:6817/1159607125","heartbeat_front_addr":"172.21.15.118:6815/1159607125","state":["exists","up"]},{"osd":2,"uuid":"b4a69006-8aa7-40ea-8410-5645691cba22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6819","nonce":3899760947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6821","nonce":3899760947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6825","nonce":3899760947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6823","nonce":3899760947}]},"public_addr":"172.21.15.118:6819/3899760947","cluster_addr":"172.21.15.118:6821/3899760947","heartbeat_back_addr":"172.21.15.118:6825/3899760947","heartbeat_front_addr":"172.21.15.118:6823/3899760947","state":["exists","up"]},{"osd":3,"uuid":"3d46b2ef-fba5-4437-b107-9a321bb94874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6801","nonce":2319016844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6803","nonce":2319016844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6807","nonce":2319016844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6805","nonce":2319016844}]},"public_addr":"172.21.15.177:6801/2319016844","cluster_addr":"172.21.15.177:6803/2319016844","heartbeat_back_addr":"172.21.15.177:6807/2319016844","heartbeat_front_addr":"172.21.15.177:6805/2319016844","state":["exists","up"]},{"osd":4,"uuid":"b56220a9-5e7e-443a-b2c2-abcbca3b0c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6809","nonce":411610750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6811","nonce":411610750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6815","nonce":411610750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6813","nonce":411610750}]},"public_addr":"172.21.15.177:6809/411610750","cluster_addr":"172.21.15.177:6811/411610750","heartbeat_back_addr":"172.21.15.177:6815/411610750","heartbeat_front_addr":"172.21.15.177:6813/411610750","state":["exists","up"]},{"osd":5,"uuid":"0fe10ad0-409a-4af7-84da-9f6eee5214bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6817","nonce":3801270167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6819","nonce":3801270167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6823","nonce":3801270167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6821","nonce":3801270167}]},"public_addr":"172.21.15.177:6817/3801270167","cluster_addr":"172.21.15.177:6819/3801270167","heartbeat_back_addr":"172.21.15.177:6823/3801270167","heartbeat_front_addr":"172.21.15.177:6821/3801270167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:04.091837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:32.064473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:01.474842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:25.475415+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:51.750854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:17.974712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/180481365":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/1224337529":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/3667900250":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6801/1066995478":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6800/180481365":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/2012557212":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/3521891639":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/287975169":"2024-03-13T03:15:57.609810+0000","172.21.15.118:6800/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/3517988861":"2024-03-13T03:17:48.514930+0000","172.21.15.118:6801/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/1647647336":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/1118679394":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/2477380828":"2024-03-13T03:15:57.609810+0000","172.21.15.118:6800/1066995478":"2024-03-13T03:16:31.649004+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:21:22.654 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f8a0406c880 msgr2=0x7f8a0406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:22.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f8a0406c880 0x7f8a0406ed40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009d10 tx=0x7f8a10009040 comp rx=0 tx=0).stop 2024-03-12T03:21:22.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a18105940 msgr2=0x7f8a18195b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:22.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a18105940 0x7f8a18195b50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800bd70 tx=0x7f8a0800be50 comp rx=0 tx=0).stop 2024-03-12T03:21:22.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 shutdown_connections 2024-03-12T03:21:22.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f8a0406c880 0x7f8a0406ed40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:22.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8a180ff940 0x7f8a18195610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:22.655 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 --2- 172.21.15.118:0/2582573535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8a18105940 0x7f8a18195b50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:22.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 >> 172.21.15.118:0/2582573535 conn(0x7f8a180fb450 msgr2=0x7f8a180fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:22.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 shutdown_connections 2024-03-12T03:21:22.656 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:22.651+0000 7f8a1fc7c700 1 -- 172.21.15.118:0/2582573535 wait complete. 2024-03-12T03:21:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: Detected new or changed devices on smithi177 2024-03-12T03:21:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:22.965 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: Adjusting osd_memory_target on smithi177 to 4970M 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:22.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:22 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2582573535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: Detected new or changed devices on smithi177 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: Adjusting osd_memory_target on smithi177 to 4970M 2024-03-12T03:21:23.091 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:21:23.092 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:22 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2582573535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:23.364 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:20:04.838424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T03:21:23.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T03:21:23.573 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:23.945 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:23 smithi118 ceph-mon[29160]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:24.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:23 smithi177 ceph-mon[37140]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:24.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:24 smithi177 ceph-mon[37140]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:24.978 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:24 smithi118 ceph-mon[29160]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 -- 172.21.15.118:0/2477077984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424106a50 msgr2=0x7fd424106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/2477077984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424106a50 0x7fd424106e30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd418009910 tx=0x7fd418009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 -- 172.21.15.118:0/2477077984 shutdown_connections 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/2477077984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd424100a50 0x7fd424100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/2477077984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424106a50 0x7fd424106e30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 -- 172.21.15.118:0/2477077984 >> 172.21.15.118:0/2477077984 conn(0x7fd4240fc540 msgr2=0x7fd4240fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 -- 172.21.15.118:0/2477077984 shutdown_connections 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 -- 172.21.15.118:0/2477077984 wait complete. 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 Processor -- start 2024-03-12T03:21:25.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.329+0000 7fd42ae1e700 1 -- start start 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd42ae1e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424100a50 0x7fd42419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd42ae1e700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd424106a50 0x7fd42419cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd42ae1e700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42419d240 con 0x7fd424100a50 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd42ae1e700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4241967b0 con 0x7fd424106a50 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd428bba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424100a50 0x7fd42419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd428bba700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424100a50 0x7fd42419c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40466/0 (socket says 172.21.15.118:40466) 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd428bba700 1 -- 172.21.15.118:0/3632109362 learned_addr learned my addr 172.21.15.118:0/3632109362 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd423fff700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd424106a50 0x7fd42419cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd428bba700 1 -- 172.21.15.118:0/3632109362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd424106a50 msgr2=0x7fd42419cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd428bba700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd424106a50 0x7fd42419cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd428bba700 1 -- 172.21.15.118:0/3632109362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4180095f0 con 0x7fd424100a50 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd428bba700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424100a50 0x7fd42419c670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd41800bd70 tx=0x7fd41800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd421ffb700 1 -- 172.21.15.118:0/3632109362 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd41801d070 con 0x7fd424100a50 2024-03-12T03:21:25.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd424196a30 con 0x7fd424100a50 2024-03-12T03:21:25.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd421ffb700 1 -- 172.21.15.118:0/3632109362 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4180056e0 con 0x7fd424100a50 2024-03-12T03:21:25.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.330+0000 7fd421ffb700 1 -- 172.21.15.118:0/3632109362 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd418017400 con 0x7fd424100a50 2024-03-12T03:21:25.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.331+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd424196f20 con 0x7fd424100a50 2024-03-12T03:21:25.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.332+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd424109050 con 0x7fd424100a50 2024-03-12T03:21:25.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.332+0000 7fd421ffb700 1 -- 172.21.15.118:0/3632109362 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd418003a60 con 0x7fd424100a50 2024-03-12T03:21:25.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.332+0000 7fd421ffb700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd41406c880 0x7fd41406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:25.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.332+0000 7fd421ffb700 1 -- 172.21.15.118:0/3632109362 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd41808ce00 con 0x7fd424100a50 2024-03-12T03:21:25.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.332+0000 7fd423fff700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd41406c880 0x7fd41406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:25.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.333+0000 7fd423fff700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd41406c880 0x7fd41406ed40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd410005950 tx=0x7fd4100058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:25.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.337+0000 7fd421ffb700 1 -- 172.21.15.118:0/3632109362 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd41805b1e0 con 0x7fd424100a50 2024-03-12T03:21:25.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.534+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd424197c70 con 0x7fd424100a50 2024-03-12T03:21:25.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.534+0000 7fd421ffb700 1 -- 172.21.15.118:0/3632109362 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd41805ad70 con 0x7fd424100a50 2024-03-12T03:21:25.537 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-12T03:21:25.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.536+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd41406c880 msgr2=0x7fd41406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:25.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.536+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd41406c880 0x7fd41406ed40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd410005950 tx=0x7fd4100058e0 comp rx=0 tx=0).stop 2024-03-12T03:21:25.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.536+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424100a50 msgr2=0x7fd42419c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:25.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.536+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424100a50 0x7fd42419c670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd41800bd70 tx=0x7fd41800be50 comp rx=0 tx=0).stop 2024-03-12T03:21:25.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.537+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 shutdown_connections 2024-03-12T03:21:25.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.537+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd41406c880 0x7fd41406ed40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:25.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.537+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd424100a50 0x7fd42419c670 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:25.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.537+0000 7fd42ae1e700 1 --2- 172.21.15.118:0/3632109362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd424106a50 0x7fd42419cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:25.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.537+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 >> 172.21.15.118:0/3632109362 conn(0x7fd4240fc540 msgr2=0x7fd4240fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:25.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.537+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 shutdown_connections 2024-03-12T03:21:25.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:25.537+0000 7fd42ae1e700 1 -- 172.21.15.118:0/3632109362 wait complete. 2024-03-12T03:21:26.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:25 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3632109362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:21:26.249 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:25 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3632109362' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:21:26.272 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T03:21:26.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:21:26.477 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:27.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:26 smithi177 ceph-mon[37140]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:27.282 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:26 smithi118 ceph-mon[29160]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 -- 172.21.15.118:0/1109573533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef880ff9d0 msgr2=0x7fef880ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 --2- 172.21.15.118:0/1109573533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef880ff9d0 0x7fef880ffdf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fef78009910 tx=0x7fef78009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 -- 172.21.15.118:0/1109573533 shutdown_connections 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 --2- 172.21.15.118:0/1109573533 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef88100bd0 0x7fef88101030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 --2- 172.21.15.118:0/1109573533 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef880ff9d0 0x7fef880ffdf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 -- 172.21.15.118:0/1109573533 >> 172.21.15.118:0/1109573533 conn(0x7fef880faf70 msgr2=0x7fef880fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 -- 172.21.15.118:0/1109573533 shutdown_connections 2024-03-12T03:21:28.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.000+0000 7fef8dee1700 1 -- 172.21.15.118:0/1109573533 wait complete. 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.001+0000 7fef8dee1700 1 Processor -- start 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.001+0000 7fef8dee1700 1 -- start start 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.001+0000 7fef8dee1700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef880ff9d0 0x7fef880699d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.001+0000 7fef8dee1700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef88100bd0 0x7fef8819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.001+0000 7fef8dee1700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8819b820 con 0x7fef88100bd0 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.001+0000 7fef8dee1700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8819b960 con 0x7fef880ff9d0 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef86ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef88100bd0 0x7fef8819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef86ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef88100bd0 0x7fef8819b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40486/0 (socket says 172.21.15.118:40486) 2024-03-12T03:21:28.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef86ffd700 1 -- 172.21.15.118:0/790090172 learned_addr learned my addr 172.21.15.118:0/790090172 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef86ffd700 1 -- 172.21.15.118:0/790090172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef880ff9d0 msgr2=0x7fef880699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef877fe700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef880ff9d0 0x7fef880699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef86ffd700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef880ff9d0 0x7fef880699d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef86ffd700 1 -- 172.21.15.118:0/790090172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef780095f0 con 0x7fef88100bd0 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef86ffd700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef88100bd0 0x7fef8819b2e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00d8d0 tx=0x7fef7c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef84ff9700 1 -- 172.21.15.118:0/790090172 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef7c009940 con 0x7fef88100bd0 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef84ff9700 1 -- 172.21.15.118:0/790090172 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef7c010460 con 0x7fef88100bd0 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8819bc10 con 0x7fef88100bd0 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef84ff9700 1 -- 172.21.15.118:0/790090172 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef7c00f5f0 con 0x7fef88100bd0 2024-03-12T03:21:28.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.002+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8819c1e0 con 0x7fef88100bd0 2024-03-12T03:21:28.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.003+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8818f550 con 0x7fef88100bd0 2024-03-12T03:21:28.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.004+0000 7fef84ff9700 1 -- 172.21.15.118:0/790090172 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fef7c00f870 con 0x7fef88100bd0 2024-03-12T03:21:28.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.004+0000 7fef84ff9700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef7006c880 0x7fef7006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:28.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.004+0000 7fef84ff9700 1 -- 172.21.15.118:0/790090172 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fef7c08b990 con 0x7fef88100bd0 2024-03-12T03:21:28.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.004+0000 7fef877fe700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef7006c880 0x7fef7006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:28.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.004+0000 7fef877fe700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef7006c880 0x7fef7006ed40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef7800b5c0 tx=0x7fef78005160 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:28.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.008+0000 7fef84ff9700 1 -- 172.21.15.118:0/790090172 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fef7c059de0 con 0x7fef88100bd0 2024-03-12T03:21:28.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.281+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fef8802cf60 con 0x7fef88100bd0 2024-03-12T03:21:28.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.284+0000 7fef84ff9700 1 -- 172.21.15.118:0/790090172 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fef7c059970 con 0x7fef88100bd0 2024-03-12T03:21:28.287 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-03-12T03:21:28.287 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQA4yu9liC3YEBAAnCKlTm49DHf3LFkXPBSimw== 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef7006c880 msgr2=0x7fef7006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef7006c880 0x7fef7006ed40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fef7800b5c0 tx=0x7fef78005160 comp rx=0 tx=0).stop 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef88100bd0 msgr2=0x7fef8819b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef88100bd0 0x7fef8819b2e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00d8d0 tx=0x7fef7c00dc90 comp rx=0 tx=0).stop 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 shutdown_connections 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef7006c880 0x7fef7006ed40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fef880ff9d0 0x7fef880699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 --2- 172.21.15.118:0/790090172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef88100bd0 0x7fef8819b2e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.286+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 >> 172.21.15.118:0/790090172 conn(0x7fef880faf70 msgr2=0x7fef88104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.287+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 shutdown_connections 2024-03-12T03:21:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:28.287+0000 7fef8dee1700 1 -- 172.21.15.118:0/790090172 wait complete. 2024-03-12T03:21:28.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:28 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/790090172' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:21:28.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:28 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/790090172' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:21:28.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:28 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/790090172' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:21:28.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:28 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/790090172' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:21:28.961 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:21:28.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T03:21:28.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T03:21:29.004 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:21:29.221 INFO:teuthology.orchestra.run.smithi177.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi177/config 2024-03-12T03:21:29.681 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:29 smithi177 ceph-mon[37140]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:29 smithi118 ceph-mon[29160]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:30.862 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 -- 172.21.15.177:0/3801252352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 msgr2=0x7f4df0103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:30.863 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 --2- 172.21.15.177:0/3801252352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df0103940 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4de0009910 tx=0x7f4de0009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:30.863 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 -- 172.21.15.177:0/3801252352 shutdown_connections 2024-03-12T03:21:30.863 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 --2- 172.21.15.177:0/3801252352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4df0103e80 0x7f4df0106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:30.863 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 --2- 172.21.15.177:0/3801252352 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df0103940 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:30.863 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 -- 172.21.15.177:0/3801252352 >> 172.21.15.177:0/3801252352 conn(0x7f4df00faf60 msgr2=0x7f4df00fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:30.863 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 -- 172.21.15.177:0/3801252352 shutdown_connections 2024-03-12T03:21:30.863 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 -- 172.21.15.177:0/3801252352 wait complete. 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.860+0000 7f4df5036700 1 Processor -- start 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4df5036700 1 -- start start 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4df5036700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df01965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4df5036700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4df0103e80 0x7f4df0196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4df5036700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df0197110 con 0x7f4df0103e80 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4df5036700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df0197250 con 0x7f4df0101550 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4deed9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df01965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4deed9d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df01965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.177:40752/0 (socket says 172.21.15.177:40752) 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4deed9d700 1 -- 172.21.15.177:0/1286586126 learned_addr learned my addr 172.21.15.177:0/1286586126 (peer_addr_for_me v2:172.21.15.177:0/0) 2024-03-12T03:21:30.865 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4de659c700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4df0103e80 0x7f4df0196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4deed9d700 1 -- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4df0103e80 msgr2=0x7f4df0196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4deed9d700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4df0103e80 0x7f4df0196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4deed9d700 1 -- 172.21.15.177:0/1286586126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de00095f0 con 0x7f4df0101550 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.861+0000 7f4deed9d700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df01965b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4de0006010 tx=0x7f4de000bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.862+0000 7f4decd99700 1 -- 172.21.15.177:0/1286586126 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4de001d070 con 0x7f4df0101550 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.862+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df019bc00 con 0x7f4df0101550 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.862+0000 7f4decd99700 1 -- 172.21.15.177:0/1286586126 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4de000f460 con 0x7f4df0101550 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.862+0000 7f4decd99700 1 -- 172.21.15.177:0/1286586126 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4de0017630 con 0x7f4df0101550 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.862+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df019c0f0 con 0x7f4df0101550 2024-03-12T03:21:30.866 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.863+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df004ea80 con 0x7f4df0101550 2024-03-12T03:21:30.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.863+0000 7f4decd99700 1 -- 172.21.15.177:0/1286586126 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4de0003a60 con 0x7f4df0101550 2024-03-12T03:21:30.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.863+0000 7f4decd99700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4ddc06c880 0x7f4ddc06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:30.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.864+0000 7f4decd99700 1 -- 172.21.15.177:0/1286586126 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f4de008cdb0 con 0x7f4df0101550 2024-03-12T03:21:30.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.864+0000 7f4de659c700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4ddc06c880 0x7f4ddc06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:30.867 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.864+0000 7f4de659c700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4ddc06c880 0x7f4ddc06ed40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005950 tx=0x7f4dd800b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:30.870 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:30.869+0000 7f4decd99700 1 -- 172.21.15.177:0/1286586126 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4de005b190 con 0x7f4df0101550 2024-03-12T03:21:31.141 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.139+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f4df019c6b0 con 0x7f4df0101550 2024-03-12T03:21:31.146 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.144+0000 7f4decd99700 1 -- 172.21.15.177:0/1286586126 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7f4de005ad20 con 0x7f4df0101550 2024-03-12T03:21:31.147 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2024-03-12T03:21:31.147 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA7yu9lT5ZvCBAAa7VEDIcCvoey/T2x+z/2Jg== 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4ddc06c880 msgr2=0x7f4ddc06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4ddc06c880 0x7f4ddc06ed40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005950 tx=0x7f4dd800b410 comp rx=0 tx=0).stop 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 msgr2=0x7f4df01965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df01965b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4de0006010 tx=0x7f4de000bd70 comp rx=0 tx=0).stop 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 shutdown_connections 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4ddc06c880 0x7f4ddc06ed40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4df0101550 0x7f4df01965b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 --2- 172.21.15.177:0/1286586126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4df0103e80 0x7f4df0196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:31.149 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 >> 172.21.15.177:0/1286586126 conn(0x7f4df00faf60 msgr2=0x7f4df00fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:31.150 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 shutdown_connections 2024-03-12T03:21:31.150 INFO:teuthology.orchestra.run.smithi177.stderr:2024-03-12T03:21:31.146+0000 7f4df5036700 1 -- 172.21.15.177:0/1286586126 wait complete. 2024-03-12T03:21:31.790 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:31 smithi177 ceph-mon[37140]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:31.791 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:31 smithi177 ceph-mon[37140]: from='client.? 172.21.15.177:0/1286586126' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:21:31.791 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:31 smithi177 ceph-mon[37140]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:21:31.791 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:31 smithi177 ceph-mon[37140]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:21:31.812 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:21:31.812 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-12T03:21:31.812 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-12T03:21:31.855 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T03:21:31.855 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T03:21:31.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T03:21:31.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:31 smithi118 ceph-mon[29160]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:31.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:31 smithi118 ceph-mon[29160]: from='client.? 172.21.15.177:0/1286586126' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:21:31.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:31 smithi118 ceph-mon[29160]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:21:31.883 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:31 smithi118 ceph-mon[29160]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:21:32.069 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:33.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 -- 172.21.15.118:0/599931292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0101c30 msgr2=0x7f16a0102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:33.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 --2- 172.21.15.118:0/599931292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0101c30 0x7f16a0102010 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f169c009940 tx=0x7f169c009c50 comp rx=0 tx=0).stop 2024-03-12T03:21:33.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 -- 172.21.15.118:0/599931292 shutdown_connections 2024-03-12T03:21:33.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 --2- 172.21.15.118:0/599931292 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16a0102550 0x7f16a010aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:33.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 --2- 172.21.15.118:0/599931292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0101c30 0x7f16a0102010 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:33.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 -- 172.21.15.118:0/599931292 >> 172.21.15.118:0/599931292 conn(0x7f16a00765c0 msgr2=0x7f16a00fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:33.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 -- 172.21.15.118:0/599931292 shutdown_connections 2024-03-12T03:21:33.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 -- 172.21.15.118:0/599931292 wait complete. 2024-03-12T03:21:33.713 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:33 smithi118 ceph-mon[29160]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:33.713 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:33 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 Processor -- start 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.710+0000 7f16a8135700 1 -- start start 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a8135700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16a0101c30 0x7f16a019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a8135700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0102550 0x7f16a019cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a8135700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a019d260 con 0x7f16a0102550 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a8135700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01967d0 con 0x7f16a0101c30 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a56d0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0102550 0x7f16a019cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:33.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a56d0700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0102550 0x7f16a019cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40514/0 (socket says 172.21.15.118:40514) 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a56d0700 1 -- 172.21.15.118:0/322078162 learned_addr learned my addr 172.21.15.118:0/322078162 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a5ed1700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16a0101c30 0x7f16a019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a56d0700 1 -- 172.21.15.118:0/322078162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16a0101c30 msgr2=0x7f16a019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a56d0700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16a0101c30 0x7f16a019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a56d0700 1 -- 172.21.15.118:0/322078162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c0095f0 con 0x7f16a0102550 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f16a56d0700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0102550 0x7f16a019cbd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f169000c8f0 tx=0x7f169000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.711+0000 7f1696ffd700 1 -- 172.21.15.118:0/322078162 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1690007a50 con 0x7f16a0102550 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.712+0000 7f1696ffd700 1 -- 172.21.15.118:0/322078162 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1690007bb0 con 0x7f16a0102550 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.712+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a0196ab0 con 0x7f16a0102550 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.712+0000 7f1696ffd700 1 -- 172.21.15.118:0/322078162 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1690018630 con 0x7f16a0102550 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.712+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a0197000 con 0x7f16a0102550 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.713+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a0105350 con 0x7f16a0102550 2024-03-12T03:21:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.713+0000 7f1696ffd700 1 -- 172.21.15.118:0/322078162 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1690007d20 con 0x7f16a0102550 2024-03-12T03:21:33.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.713+0000 7f1696ffd700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f168c06c880 0x7f168c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:33.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.714+0000 7f16a5ed1700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f168c06c880 0x7f168c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:33.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.714+0000 7f1696ffd700 1 -- 172.21.15.118:0/322078162 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f169008b920 con 0x7f16a0102550 2024-03-12T03:21:33.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.714+0000 7f16a5ed1700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f168c06c880 0x7f168c06ed40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f169c00b870 tx=0x7f169c005c00 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:33.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.718+0000 7f1696ffd700 1 -- 172.21.15.118:0/322078162 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1690059d00 con 0x7f16a0102550 2024-03-12T03:21:33.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:33 smithi177 ceph-mon[37140]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:33 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:33.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.965+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f16a0068c80 con 0x7f16a0102550 2024-03-12T03:21:33.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.969+0000 7f1696ffd700 1 -- 172.21.15.118:0/322078162 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+173475 (secure 0 0 0) 0x7f1690059890 con 0x7f16a0102550 2024-03-12T03:21:33.974 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f168c06c880 msgr2=0x7f168c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f168c06c880 0x7f168c06ed40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f169c00b870 tx=0x7f169c005c00 comp rx=0 tx=0).stop 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0102550 msgr2=0x7f16a019cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0102550 0x7f16a019cbd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f169000c8f0 tx=0x7f169000ccb0 comp rx=0 tx=0).stop 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 shutdown_connections 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f168c06c880 0x7f168c06ed40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f16a0101c30 0x7f16a019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 --2- 172.21.15.118:0/322078162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f16a0102550 0x7f16a019cbd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.971+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 >> 172.21.15.118:0/322078162 conn(0x7f16a00765c0 msgr2=0x7f16a0102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.972+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 shutdown_connections 2024-03-12T03:21:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:33.972+0000 7f16a8135700 1 -- 172.21.15.118:0/322078162 wait complete. 2024-03-12T03:21:34.673 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":19,"active_gid":14221,"active_name":"smithi118.zzzuvl","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":3827716556},{"type":"v1","addr":"172.21.15.118:6801","nonce":3827716556}]},"active_addr":"172.21.15.118:6801/3827716556","active_change":"2024-03-12T03:17:48.515087+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14242,"name":"smithi177.tmmsuz","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.118:8443/","prometheus":"http://172.21.15.118:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":319015752}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1231807437}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3138617910}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":3123764903}]}]} 2024-03-12T03:21:34.678 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T03:21:34.678 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T03:21:34.678 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:21:34.927 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:35.049 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:34 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/322078162' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:21:35.049 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:34 smithi118 ceph-mon[29160]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:35.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:34 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/322078162' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:21:35.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:34 smithi177 ceph-mon[37140]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:36.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 -- 172.21.15.118:0/41142016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 msgr2=0x7fcb000ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 --2- 172.21.15.118:0/41142016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb000ff830 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009910 tx=0x7fcaf4009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 -- 172.21.15.118:0/41142016 shutdown_connections 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 --2- 172.21.15.118:0/41142016 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb000ffe00 0x7fcb001056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 --2- 172.21.15.118:0/41142016 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb000ff830 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 -- 172.21.15.118:0/41142016 >> 172.21.15.118:0/41142016 conn(0x7fcb00075da0 msgr2=0x7fcb000761b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 -- 172.21.15.118:0/41142016 shutdown_connections 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.522+0000 7fcb0643f700 1 -- 172.21.15.118:0/41142016 wait complete. 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcb0643f700 1 Processor -- start 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcb0643f700 1 -- start start 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcb0643f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb001003d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcb0643f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb000ffe00 0x7fcb00100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:36.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcb0643f700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb00100e50 con 0x7fcb000ff450 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcb0643f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb00100f90 con 0x7fcb000ffe00 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcafffff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb001003d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcafffff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb001003d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:54218/0 (socket says 172.21.15.118:54218) 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.523+0000 7fcafffff700 1 -- 172.21.15.118:0/1073372222 learned_addr learned my addr 172.21.15.118:0/1073372222 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcafffff700 1 -- 172.21.15.118:0/1073372222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb000ffe00 msgr2=0x7fcb00100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcaff7fe700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb000ffe00 0x7fcb00100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcafffff700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb000ffe00 0x7fcb00100910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcafffff700 1 -- 172.21.15.118:0/1073372222 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf40095f0 con 0x7fcb000ff450 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcafffff700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb001003d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4006010 tx=0x7fcaf400be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcafd7fa700 1 -- 172.21.15.118:0/1073372222 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf401d070 con 0x7fcb000ff450 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcafd7fa700 1 -- 172.21.15.118:0/1073372222 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcaf40056e0 con 0x7fcb000ff450 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb00101210 con 0x7fcb000ff450 2024-03-12T03:21:36.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcafd7fa700 1 -- 172.21.15.118:0/1073372222 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcaf4017400 con 0x7fcb000ff450 2024-03-12T03:21:36.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.524+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb001a2dc0 con 0x7fcb000ff450 2024-03-12T03:21:36.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.525+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb00109030 con 0x7fcb000ff450 2024-03-12T03:21:36.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.525+0000 7fcafd7fa700 1 -- 172.21.15.118:0/1073372222 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcaf400f860 con 0x7fcb000ff450 2024-03-12T03:21:36.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.526+0000 7fcafd7fa700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcae806c880 0x7fcae806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:36.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.526+0000 7fcafd7fa700 1 -- 172.21.15.118:0/1073372222 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fcaf408cce0 con 0x7fcb000ff450 2024-03-12T03:21:36.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.526+0000 7fcaff7fe700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcae806c880 0x7fcae806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:36.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.526+0000 7fcaff7fe700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcae806c880 0x7fcae806ed40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0005950 tx=0x7fcaf00058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:36.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.530+0000 7fcafd7fa700 1 -- 172.21.15.118:0/1073372222 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcaf40576f0 con 0x7fcb000ff450 2024-03-12T03:21:36.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.725+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb00068c60 con 0x7fcb000ff450 2024-03-12T03:21:36.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.726+0000 7fcafd7fa700 1 -- 172.21.15.118:0/1073372222 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11387 (secure 0 0 0) 0x7fcaf405ad10 con 0x7fcb000ff450 2024-03-12T03:21:36.728 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:21:36.728 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":41,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","created":"2024-03-12T03:15:23.858621+0000","modified":"2024-03-12T03:21:20.621261+0000","last_up_change":"2024-03-12T03:21:19.616038+0000","last_in_change":"2024-03-12T03:21:00.310107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:20:04.838424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96b74c65-8c14-4178-9e8d-b981032baa7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6803","nonce":1275003428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6805","nonce":1275003428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6809","nonce":1275003428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6807","nonce":1275003428}]},"public_addr":"172.21.15.118:6803/1275003428","cluster_addr":"172.21.15.118:6805/1275003428","heartbeat_back_addr":"172.21.15.118:6809/1275003428","heartbeat_front_addr":"172.21.15.118:6807/1275003428","state":["exists","up"]},{"osd":1,"uuid":"3c911034-856e-43e7-b267-2a6c63cf46fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6811","nonce":1159607125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6813","nonce":1159607125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6817","nonce":1159607125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6815","nonce":1159607125}]},"public_addr":"172.21.15.118:6811/1159607125","cluster_addr":"172.21.15.118:6813/1159607125","heartbeat_back_addr":"172.21.15.118:6817/1159607125","heartbeat_front_addr":"172.21.15.118:6815/1159607125","state":["exists","up"]},{"osd":2,"uuid":"b4a69006-8aa7-40ea-8410-5645691cba22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6819","nonce":3899760947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6821","nonce":3899760947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6825","nonce":3899760947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6823","nonce":3899760947}]},"public_addr":"172.21.15.118:6819/3899760947","cluster_addr":"172.21.15.118:6821/3899760947","heartbeat_back_addr":"172.21.15.118:6825/3899760947","heartbeat_front_addr":"172.21.15.118:6823/3899760947","state":["exists","up"]},{"osd":3,"uuid":"3d46b2ef-fba5-4437-b107-9a321bb94874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6801","nonce":2319016844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6803","nonce":2319016844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6807","nonce":2319016844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6805","nonce":2319016844}]},"public_addr":"172.21.15.177:6801/2319016844","cluster_addr":"172.21.15.177:6803/2319016844","heartbeat_back_addr":"172.21.15.177:6807/2319016844","heartbeat_front_addr":"172.21.15.177:6805/2319016844","state":["exists","up"]},{"osd":4,"uuid":"b56220a9-5e7e-443a-b2c2-abcbca3b0c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6809","nonce":411610750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6811","nonce":411610750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6815","nonce":411610750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6813","nonce":411610750}]},"public_addr":"172.21.15.177:6809/411610750","cluster_addr":"172.21.15.177:6811/411610750","heartbeat_back_addr":"172.21.15.177:6815/411610750","heartbeat_front_addr":"172.21.15.177:6813/411610750","state":["exists","up"]},{"osd":5,"uuid":"0fe10ad0-409a-4af7-84da-9f6eee5214bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6817","nonce":3801270167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6819","nonce":3801270167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6823","nonce":3801270167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6821","nonce":3801270167}]},"public_addr":"172.21.15.177:6817/3801270167","cluster_addr":"172.21.15.177:6819/3801270167","heartbeat_back_addr":"172.21.15.177:6823/3801270167","heartbeat_front_addr":"172.21.15.177:6821/3801270167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:04.091837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:32.064473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:01.474842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:25.475415+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:51.750854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:17.974712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/180481365":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/1224337529":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/3667900250":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6801/1066995478":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6800/180481365":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/2012557212":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/3521891639":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/287975169":"2024-03-13T03:15:57.609810+0000","172.21.15.118:6800/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/3517988861":"2024-03-13T03:17:48.514930+0000","172.21.15.118:6801/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/1647647336":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/1118679394":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/2477380828":"2024-03-13T03:15:57.609810+0000","172.21.15.118:6800/1066995478":"2024-03-13T03:16:31.649004+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:21:36.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcae806c880 msgr2=0x7fcae806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:36.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcae806c880 0x7fcae806ed40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0005950 tx=0x7fcaf00058e0 comp rx=0 tx=0).stop 2024-03-12T03:21:36.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 msgr2=0x7fcb001003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:36.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb001003d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4006010 tx=0x7fcaf400be20 comp rx=0 tx=0).stop 2024-03-12T03:21:36.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 shutdown_connections 2024-03-12T03:21:36.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcae806c880 0x7fcae806ed40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:36.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb000ff450 0x7fcb001003d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:36.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 --2- 172.21.15.118:0/1073372222 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb000ffe00 0x7fcb00100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:36.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 >> 172.21.15.118:0/1073372222 conn(0x7fcb00075da0 msgr2=0x7fcb000fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:36.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 shutdown_connections 2024-03-12T03:21:36.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:36.728+0000 7fcb0643f700 1 -- 172.21.15.118:0/1073372222 wait complete. 2024-03-12T03:21:37.420 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T03:21:37.421 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:21:37.628 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:37 smithi177 ceph-mon[37140]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:37 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1073372222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:37.967 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:37 smithi118 ceph-mon[29160]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:37.967 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:37 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1073372222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:39.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 -- 172.21.15.118:0/4292432810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc1059e0 msgr2=0x7f05fc105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 --2- 172.21.15.118:0/4292432810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc1059e0 0x7f05fc105dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f05f400b0a0 tx=0x7f05f400b3b0 comp rx=0 tx=0).stop 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 -- 172.21.15.118:0/4292432810 shutdown_connections 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 --2- 172.21.15.118:0/4292432810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05fc0ff9e0 0x7f05fc0ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 --2- 172.21.15.118:0/4292432810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc1059e0 0x7f05fc105dc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 -- 172.21.15.118:0/4292432810 >> 172.21.15.118:0/4292432810 conn(0x7f05fc0fb4d0 msgr2=0x7f05fc0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 -- 172.21.15.118:0/4292432810 shutdown_connections 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.205+0000 7f06016a9700 1 -- 172.21.15.118:0/4292432810 wait complete. 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f06016a9700 1 Processor -- start 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f06016a9700 1 -- start start 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f06016a9700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff9e0 0x7f05fc19b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f06016a9700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05fc1059e0 0x7f05fc19bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f06016a9700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc19c140 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f06016a9700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc195720 con 0x7f05fc1059e0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f05faffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff9e0 0x7f05fc19b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f05faffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff9e0 0x7f05fc19b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:54238/0 (socket says 172.21.15.118:54238) 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f05faffd700 1 -- 172.21.15.118:0/2815129204 learned_addr learned my addr 172.21.15.118:0/2815129204 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.206+0000 7f05fa7fc700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05fc1059e0 0x7f05fc19bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f05faffd700 1 -- 172.21.15.118:0/2815129204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05fc1059e0 msgr2=0x7f05fc19bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f05faffd700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05fc1059e0 0x7f05fc19bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f05faffd700 1 -- 172.21.15.118:0/2815129204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05f4009d00 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f05faffd700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff9e0 0x7f05fc19b5e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05f4000f80 tx=0x7f05f4007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f05f3fff700 1 -- 172.21.15.118:0/2815129204 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05f400e040 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f05f3fff700 1 -- 172.21.15.118:0/2815129204 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f05f40096a0 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc1959a0 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f05f3fff700 1 -- 172.21.15.118:0/2815129204 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05f401d6b0 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.207+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc195e90 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.208+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc107fe0 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.208+0000 7f05f3fff700 1 -- 172.21.15.118:0/2815129204 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f05f4003c10 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.209+0000 7f05f3fff700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f05e806c880 0x7f05e806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:39.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.209+0000 7f05fa7fc700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f05e806c880 0x7f05e806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:39.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.209+0000 7f05f3fff700 1 -- 172.21.15.118:0/2815129204 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f05f408cee0 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.209+0000 7f05fa7fc700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f05e806c880 0x7f05e806ed40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05fc196e30 tx=0x7f05e400b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:39.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.213+0000 7f05f3fff700 1 -- 172.21.15.118:0/2815129204 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f05f405b340 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.411+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05fc101120 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.411+0000 7f05f3fff700 1 -- 172.21.15.118:0/2815129204 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11387 (secure 0 0 0) 0x7f05f4017090 con 0x7f05fc0ff9e0 2024-03-12T03:21:39.414 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:21:39.414 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":41,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","created":"2024-03-12T03:15:23.858621+0000","modified":"2024-03-12T03:21:20.621261+0000","last_up_change":"2024-03-12T03:21:19.616038+0000","last_in_change":"2024-03-12T03:21:00.310107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:20:04.838424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96b74c65-8c14-4178-9e8d-b981032baa7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6803","nonce":1275003428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6805","nonce":1275003428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6809","nonce":1275003428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6807","nonce":1275003428}]},"public_addr":"172.21.15.118:6803/1275003428","cluster_addr":"172.21.15.118:6805/1275003428","heartbeat_back_addr":"172.21.15.118:6809/1275003428","heartbeat_front_addr":"172.21.15.118:6807/1275003428","state":["exists","up"]},{"osd":1,"uuid":"3c911034-856e-43e7-b267-2a6c63cf46fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6811","nonce":1159607125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6813","nonce":1159607125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6817","nonce":1159607125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6815","nonce":1159607125}]},"public_addr":"172.21.15.118:6811/1159607125","cluster_addr":"172.21.15.118:6813/1159607125","heartbeat_back_addr":"172.21.15.118:6817/1159607125","heartbeat_front_addr":"172.21.15.118:6815/1159607125","state":["exists","up"]},{"osd":2,"uuid":"b4a69006-8aa7-40ea-8410-5645691cba22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6819","nonce":3899760947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6821","nonce":3899760947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6825","nonce":3899760947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6823","nonce":3899760947}]},"public_addr":"172.21.15.118:6819/3899760947","cluster_addr":"172.21.15.118:6821/3899760947","heartbeat_back_addr":"172.21.15.118:6825/3899760947","heartbeat_front_addr":"172.21.15.118:6823/3899760947","state":["exists","up"]},{"osd":3,"uuid":"3d46b2ef-fba5-4437-b107-9a321bb94874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6801","nonce":2319016844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6803","nonce":2319016844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6807","nonce":2319016844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6805","nonce":2319016844}]},"public_addr":"172.21.15.177:6801/2319016844","cluster_addr":"172.21.15.177:6803/2319016844","heartbeat_back_addr":"172.21.15.177:6807/2319016844","heartbeat_front_addr":"172.21.15.177:6805/2319016844","state":["exists","up"]},{"osd":4,"uuid":"b56220a9-5e7e-443a-b2c2-abcbca3b0c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6809","nonce":411610750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6811","nonce":411610750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6815","nonce":411610750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6813","nonce":411610750}]},"public_addr":"172.21.15.177:6809/411610750","cluster_addr":"172.21.15.177:6811/411610750","heartbeat_back_addr":"172.21.15.177:6815/411610750","heartbeat_front_addr":"172.21.15.177:6813/411610750","state":["exists","up"]},{"osd":5,"uuid":"0fe10ad0-409a-4af7-84da-9f6eee5214bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6817","nonce":3801270167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6819","nonce":3801270167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6823","nonce":3801270167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6821","nonce":3801270167}]},"public_addr":"172.21.15.177:6817/3801270167","cluster_addr":"172.21.15.177:6819/3801270167","heartbeat_back_addr":"172.21.15.177:6823/3801270167","heartbeat_front_addr":"172.21.15.177:6821/3801270167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:04.091837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:32.064473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:01.474842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:25.475415+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:51.750854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:17.974712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/180481365":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/1224337529":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/3667900250":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6801/1066995478":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6800/180481365":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/2012557212":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/3521891639":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/287975169":"2024-03-13T03:15:57.609810+0000","172.21.15.118:6800/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/3517988861":"2024-03-13T03:17:48.514930+0000","172.21.15.118:6801/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/1647647336":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/1118679394":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/2477380828":"2024-03-13T03:15:57.609810+0000","172.21.15.118:6800/1066995478":"2024-03-13T03:16:31.649004+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:21:39.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f05e806c880 msgr2=0x7f05e806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:39.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f05e806c880 0x7f05e806ed40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f05fc196e30 tx=0x7f05e400b410 comp rx=0 tx=0).stop 2024-03-12T03:21:39.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff9e0 msgr2=0x7f05fc19b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:39.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff9e0 0x7f05fc19b5e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f05f4000f80 tx=0x7f05f4007b20 comp rx=0 tx=0).stop 2024-03-12T03:21:39.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 shutdown_connections 2024-03-12T03:21:39.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f05e806c880 0x7f05e806ed40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:39.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05fc0ff9e0 0x7f05fc19b5e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:39.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 --2- 172.21.15.118:0/2815129204 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f05fc1059e0 0x7f05fc19bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:39.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.413+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 >> 172.21.15.118:0/2815129204 conn(0x7f05fc0fb4d0 msgr2=0x7f05fc0fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:39.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.414+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 shutdown_connections 2024-03-12T03:21:39.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:39.414+0000 7f06016a9700 1 -- 172.21.15.118:0/2815129204 wait complete. 2024-03-12T03:21:39.745 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:39 smithi118 ceph-mon[29160]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:39.745 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:39 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2815129204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:39.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:39 smithi177 ceph-mon[37140]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:39.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:39 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2815129204' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:21:40.135 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T03:21:40.135 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T03:21:40.135 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T03:21:40.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-12T03:21:40.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-12T03:21:40.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-12T03:21:40.394 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:40.402 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:41.774 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:41 smithi118 ceph-mon[29160]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:41.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:41 smithi177 ceph-mon[37140]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:42.205 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:42.210 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:42.325 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:43.688 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:43 smithi118 ceph-mon[29160]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:43.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:43 smithi177 ceph-mon[37140]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 -- 172.21.15.118:0/3920972547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428102580 msgr2=0x7fa42810aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 --2- 172.21.15.118:0/3920972547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428102580 0x7fa42810aa70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009910 tx=0x7fa41c009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 -- 172.21.15.118:0/3920972547 shutdown_connections 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 --2- 172.21.15.118:0/3920972547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428102580 0x7fa42810aa70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 --2- 172.21.15.118:0/3920972547 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa428101c60 0x7fa428102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 -- 172.21.15.118:0/3920972547 >> 172.21.15.118:0/3920972547 conn(0x7fa4280fb450 msgr2=0x7fa4280fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 -- 172.21.15.118:0/3920972547 shutdown_connections 2024-03-12T03:21:44.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.206+0000 7fa42e670700 1 -- 172.21.15.118:0/3920972547 wait complete. 2024-03-12T03:21:44.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa42e670700 1 Processor -- start 2024-03-12T03:21:44.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa42e670700 1 -- start start 2024-03-12T03:21:44.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa42e670700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428101c60 0x7fa42819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:44.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa42e670700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa428102580 0x7fa42819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:44.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa42e670700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa42819d230 con 0x7fa428101c60 2024-03-12T03:21:44.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa42e670700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa428196800 con 0x7fa428102580 2024-03-12T03:21:44.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa427fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428101c60 0x7fa42819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa427fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428101c60 0x7fa42819c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:54250/0 (socket says 172.21.15.118:54250) 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa427fff700 1 -- 172.21.15.118:0/1393657810 learned_addr learned my addr 172.21.15.118:0/1393657810 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa4277fe700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa428102580 0x7fa42819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa427fff700 1 -- 172.21.15.118:0/1393657810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa428102580 msgr2=0x7fa42819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa427fff700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa428102580 0x7fa42819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.207+0000 7fa427fff700 1 -- 172.21.15.118:0/1393657810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa41c0095f0 con 0x7fa428101c60 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.208+0000 7fa427fff700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428101c60 0x7fa42819c6a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa41800c930 tx=0x7fa41800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.208+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa418007ad0 con 0x7fa428101c60 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.208+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa41800ce80 con 0x7fa428101c60 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.208+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa418018630 con 0x7fa428101c60 2024-03-12T03:21:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.208+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa428196ae0 con 0x7fa428101c60 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.208+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa428197030 con 0x7fa428101c60 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.209+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa42810df40 con 0x7fa428101c60 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.209+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fa418007c30 con 0x7fa428101c60 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa4257fa700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fa41006c880 0x7fa41006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fa41808b850 con 0x7fa428101c60 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa4277fe700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fa41006c880 0x7fa41006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa4257fa700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] conn(0x7fa410072560 0x7fa410074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 --> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa410075030 con 0x7fa410072560 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4180902a0 con 0x7fa428101c60 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa42cc0d700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] conn(0x7fa410072560 0x7fa410074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa4277fe700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fa41006c880 0x7fa41006ed40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa428197f30 tx=0x7fa41c01a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.210+0000 7fa42cc0d700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] conn(0x7fa410072560 0x7fa410074980 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:44.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.211+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== osd.2 v2:172.21.15.118:6818/3899760947 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa410075030 con 0x7fa410072560 2024-03-12T03:21:44.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 --> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa428069510 con 0x7fa410072560 2024-03-12T03:21:44.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa4257fa700 1 -- 172.21.15.118:0/1393657810 <== osd.2 v2:172.21.15.118:6818/3899760947 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa428069510 con 0x7fa410072560 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] conn(0x7fa410072560 msgr2=0x7fa410074980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa42e670700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] conn(0x7fa410072560 0x7fa410074980 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fa41006c880 msgr2=0x7fa41006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa42e670700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fa41006c880 0x7fa41006ed40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa428197f30 tx=0x7fa41c01a040 comp rx=0 tx=0).stop 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428101c60 msgr2=0x7fa42819c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.227+0000 7fa42e670700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428101c60 0x7fa42819c6a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa41800c930 tx=0x7fa41800cc40 comp rx=0 tx=0).stop 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 shutdown_connections 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fa41006c880 0x7fa41006ed40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa428101c60 0x7fa42819c6a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.118:6818/3899760947,v1:172.21.15.118:6819/3899760947] conn(0x7fa410072560 0x7fa410074980 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 --2- 172.21.15.118:0/1393657810 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fa428102580 0x7fa42819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:44.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 >> 172.21.15.118:0/1393657810 conn(0x7fa4280fb450 msgr2=0x7fa428102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:44.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 shutdown_connections 2024-03-12T03:21:44.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:44.228+0000 7fa42e670700 1 -- 172.21.15.118:0/1393657810 wait complete. 2024-03-12T03:21:45.029 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:44 smithi177 ceph-mon[37140]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:45.197 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:44 smithi118 ceph-mon[29160]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:45.649 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:46.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 -- 172.21.15.118:0/3739636361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0102570 msgr2=0x7fcea010aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:46.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 --2- 172.21.15.118:0/3739636361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0102570 0x7fcea010aa60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fce88009940 tx=0x7fce88009c50 comp rx=0 tx=0).stop 2024-03-12T03:21:46.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 -- 172.21.15.118:0/3739636361 shutdown_connections 2024-03-12T03:21:46.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 --2- 172.21.15.118:0/3739636361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0102570 0x7fcea010aa60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 --2- 172.21.15.118:0/3739636361 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcea0101c50 0x7fcea0102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 -- 172.21.15.118:0/3739636361 >> 172.21.15.118:0/3739636361 conn(0x7fcea00765c0 msgr2=0x7fcea00fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:46.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 -- 172.21.15.118:0/3739636361 shutdown_connections 2024-03-12T03:21:46.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 -- 172.21.15.118:0/3739636361 wait complete. 2024-03-12T03:21:46.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.531+0000 7fcea4b90700 1 Processor -- start 2024-03-12T03:21:46.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fcea4b90700 1 -- start start 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fcea4b90700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0101c50 0x7fcea019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fcea4b90700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcea0102570 0x7fcea019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fcea4b90700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea019d2f0 con 0x7fcea0101c50 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fcea4b90700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea01968d0 con 0x7fcea0102570 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce9e59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0101c50 0x7fcea019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce9e59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0101c50 0x7fcea019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37954/0 (socket says 172.21.15.118:37954) 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce9e59c700 1 -- 172.21.15.118:0/2570117082 learned_addr learned my addr 172.21.15.118:0/2570117082 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce95d9b700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcea0102570 0x7fcea019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce9e59c700 1 -- 172.21.15.118:0/2570117082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcea0102570 msgr2=0x7fcea019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce9e59c700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcea0102570 0x7fcea019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce9e59c700 1 -- 172.21.15.118:0/2570117082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880095f0 con 0x7fcea0101c50 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.532+0000 7fce9e59c700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0101c50 0x7fcea019c790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce9000b4b0 tx=0x7fce9000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:46.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.533+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce90011800 con 0x7fcea0101c50 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.533+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce90011e20 con 0x7fcea0101c50 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.533+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce9000f2e0 con 0x7fcea0101c50 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.533+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea0196bb0 con 0x7fcea0101c50 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.533+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01970d0 con 0x7fcea0101c50 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.533+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcea010dfb0 con 0x7fcea0101c50 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.534+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fce90010380 con 0x7fcea0101c50 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.534+0000 7fce97fff700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fce8c06c880 0x7fce8c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:46.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fce9008b4a0 con 0x7fcea0101c50 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce95d9b700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fce8c06c880 0x7fce8c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce97fff700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] conn(0x7fce8c072560 0x7fce8c074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 --> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce8c075030 con 0x7fce8c072560 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce9008db70 con 0x7fcea0101c50 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce9ed9d700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] conn(0x7fce8c072560 0x7fce8c074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce95d9b700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fce8c06c880 0x7fce8c06ed40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce88009910 tx=0x7fce8800b8d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.535+0000 7fce9ed9d700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] conn(0x7fce8c072560 0x7fce8c074980 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:46.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.536+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== osd.0 v2:172.21.15.118:6802/1275003428 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fce8c075030 con 0x7fce8c072560 2024-03-12T03:21:46.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 --> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcea0197590 con 0x7fce8c072560 2024-03-12T03:21:46.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fce97fff700 1 -- 172.21.15.118:0/2570117082 <== osd.0 v2:172.21.15.118:6802/1275003428 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcea0197590 con 0x7fce8c072560 2024-03-12T03:21:46.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] conn(0x7fce8c072560 msgr2=0x7fce8c074980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] conn(0x7fce8c072560 0x7fce8c074980 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fce8c06c880 msgr2=0x7fce8c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fce8c06c880 0x7fce8c06ed40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce88009910 tx=0x7fce8800b8d0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0101c50 msgr2=0x7fcea019c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0101c50 0x7fcea019c790 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fce9000b4b0 tx=0x7fce9000b7c0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 shutdown_connections 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6802/1275003428,v1:172.21.15.118:6803/1275003428] conn(0x7fce8c072560 0x7fce8c074980 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fce8c06c880 0x7fce8c06ed40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcea0101c50 0x7fcea019c790 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 --2- 172.21.15.118:0/2570117082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcea0102570 0x7fcea019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 >> 172.21.15.118:0/2570117082 conn(0x7fcea00765c0 msgr2=0x7fcea0102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.551+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 shutdown_connections 2024-03-12T03:21:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:46.552+0000 7fcea4b90700 1 -- 172.21.15.118:0/2570117082 wait complete. 2024-03-12T03:21:47.343 INFO:teuthology.orchestra.run.smithi118.stdout:38654705698 2024-03-12T03:21:47.343 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T03:21:47.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 -- 172.21.15.118:0/2264162625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 msgr2=0x7f82e4105d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:47.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 --2- 172.21.15.118:0/2264162625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e4105d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009910 tx=0x7f82e0009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 -- 172.21.15.118:0/2264162625 shutdown_connections 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 --2- 172.21.15.118:0/2264162625 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82e40ff970 0x7f82e40ffdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 --2- 172.21.15.118:0/2264162625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e4105d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 -- 172.21.15.118:0/2264162625 >> 172.21.15.118:0/2264162625 conn(0x7f82e40fb460 msgr2=0x7f82e40fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 -- 172.21.15.118:0/2264162625 shutdown_connections 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 -- 172.21.15.118:0/2264162625 wait complete. 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.809+0000 7f82ec46f700 1 Processor -- start 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82ec46f700 1 -- start start 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82ec46f700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82e40ff970 0x7f82e40fbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82ec46f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e40fc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82ec46f700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e40fc660 con 0x7f82e4105970 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82ec46f700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e40fc7a0 con 0x7f82e40ff970 2024-03-12T03:21:47.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82e9a0a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e40fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82e9a0a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e40fc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37962/0 (socket says 172.21.15.118:37962) 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82e9a0a700 1 -- 172.21.15.118:0/400726211 learned_addr learned my addr 172.21.15.118:0/400726211 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82ea20b700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82e40ff970 0x7f82e40fbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82e9a0a700 1 -- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82e40ff970 msgr2=0x7f82e40fbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82e9a0a700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82e40ff970 0x7f82e40fbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82e9a0a700 1 -- 172.21.15.118:0/400726211 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e00095f0 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82e9a0a700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e40fc120 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82d400d8d0 tx=0x7f82d400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82d40098a0 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82d4010460 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e41a2b00 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.810+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82d400f5f0 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.811+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e41a2e60 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.811+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82e410ce70 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.812+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f82d4009a00 con 0x7f82e4105970 2024-03-12T03:21:47.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.812+0000 7f82db7fe700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f82d006c880 0x7f82d006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.812+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f82d408bd30 con 0x7f82e4105970 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.812+0000 7f82ea20b700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f82d006c880 0x7f82d006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.812+0000 7f82db7fe700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] conn(0x7f82d0072560 0x7f82d0074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.812+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 --> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82d0075030 con 0x7f82d0072560 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.812+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82d408c0f0 con 0x7f82e4105970 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.813+0000 7f82ea20b700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f82d006c880 0x7f82d006ed40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f82e0000c00 tx=0x7f82e001a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.813+0000 7f82eaa0c700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] conn(0x7f82d0072560 0x7f82d0074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.813+0000 7f82eaa0c700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] conn(0x7f82d0072560 0x7f82d0074980 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.814+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== osd.4 v2:172.21.15.177:6808/411610750 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f82d0075030 con 0x7f82d0072560 2024-03-12T03:21:47.829 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:47 smithi118 ceph-mon[29160]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:47.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.830+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 --> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f82e4061470 con 0x7f82d0072560 2024-03-12T03:21:47.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.830+0000 7f82db7fe700 1 -- 172.21.15.118:0/400726211 <== osd.4 v2:172.21.15.177:6808/411610750 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f82e4061470 con 0x7f82d0072560 2024-03-12T03:21:47.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] conn(0x7f82d0072560 msgr2=0x7f82d0074980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:47.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] conn(0x7f82d0072560 0x7f82d0074980 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f82d006c880 msgr2=0x7f82d006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f82d006c880 0x7f82d006ed40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f82e0000c00 tx=0x7f82e001a040 comp rx=0 tx=0).stop 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 msgr2=0x7f82e40fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e40fc120 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f82d400d8d0 tx=0x7f82d400dbe0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 shutdown_connections 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f82d006c880 0x7f82d006ed40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f82e40ff970 0x7f82e40fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.177:6808/411610750,v1:172.21.15.177:6809/411610750] conn(0x7f82d0072560 0x7f82d0074980 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 --2- 172.21.15.118:0/400726211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82e4105970 0x7f82e40fc120 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 >> 172.21.15.118:0/400726211 conn(0x7f82e40fb460 msgr2=0x7f82e40693d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 shutdown_connections 2024-03-12T03:21:47.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:47.831+0000 7f82ec46f700 1 -- 172.21.15.118:0/400726211 wait complete. 2024-03-12T03:21:47.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:47 smithi177 ceph-mon[37140]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:48.744 INFO:teuthology.orchestra.run.smithi118.stdout:81604378648 2024-03-12T03:21:48.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T03:21:48.844 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:48.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:21:49.147 INFO:teuthology.orchestra.run.smithi118.stdout:150323855373 2024-03-12T03:21:49.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-12T03:21:49.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 -- 172.21.15.118:0/2382096293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24100a50 msgr2=0x7fec24100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 --2- 172.21.15.118:0/2382096293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24100a50 0x7fec24100ed0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fec14009910 tx=0x7fec14009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 -- 172.21.15.118:0/2382096293 shutdown_connections 2024-03-12T03:21:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 --2- 172.21.15.118:0/2382096293 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24100a50 0x7fec24100ed0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 --2- 172.21.15.118:0/2382096293 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec24106a50 0x7fec24106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 -- 172.21.15.118:0/2382096293 >> 172.21.15.118:0/2382096293 conn(0x7fec240fc540 msgr2=0x7fec240fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 -- 172.21.15.118:0/2382096293 shutdown_connections 2024-03-12T03:21:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.339+0000 7fec2b649700 1 -- 172.21.15.118:0/2382096293 wait complete. 2024-03-12T03:21:49.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec2b649700 1 Processor -- start 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec2b649700 1 -- start start 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec2b649700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec24100a50 0x7fec241945b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec2b649700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24106a50 0x7fec24194af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec2b649700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec24195030 con 0x7fec24106a50 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec2b649700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec241951a0 con 0x7fec24100a50 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec28be4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24106a50 0x7fec24194af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec28be4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24106a50 0x7fec24194af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37978/0 (socket says 172.21.15.118:37978) 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec28be4700 1 -- 172.21.15.118:0/3029969737 learned_addr learned my addr 172.21.15.118:0/3029969737 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.340+0000 7fec293e5700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec24100a50 0x7fec241945b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec28be4700 1 -- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec24100a50 msgr2=0x7fec241945b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec28be4700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec24100a50 0x7fec241945b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec28be4700 1 -- 172.21.15.118:0/3029969737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec140095f0 con 0x7fec24106a50 2024-03-12T03:21:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec28be4700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24106a50 0x7fec24194af0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fec1400bd70 tx=0x7fec1400bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1401d070 con 0x7fec24106a50 2024-03-12T03:21:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec24195420 con 0x7fec24106a50 2024-03-12T03:21:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec140056e0 con 0x7fec24106a50 2024-03-12T03:21:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec241a7210 con 0x7fec24106a50 2024-03-12T03:21:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.341+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fec1400fe40 con 0x7fec24106a50 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.342+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec2410df10 con 0x7fec24106a50 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.343+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fec140174b0 con 0x7fec24106a50 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.343+0000 7fec1a7fc700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fec1006c880 0x7fec1006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.343+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fec1408caa0 con 0x7fec24106a50 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.343+0000 7fec293e5700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fec1006c880 0x7fec1006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.343+0000 7fec1a7fc700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] conn(0x7fec10072560 0x7fec10074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.343+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 --> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec10075030 con 0x7fec10072560 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.343+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec140912a0 con 0x7fec24106a50 2024-03-12T03:21:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.344+0000 7fec293e5700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fec1006c880 0x7fec1006ed40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec1c005950 tx=0x7fec1c00b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:49.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.344+0000 7fec29be6700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] conn(0x7fec10072560 0x7fec10074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:49.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.344+0000 7fec29be6700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] conn(0x7fec10072560 0x7fec10074980 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:49.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.345+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== osd.5 v2:172.21.15.177:6816/3801270167 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fec10075030 con 0x7fec10072560 2024-03-12T03:21:49.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.361+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 --> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fec24061430 con 0x7fec10072560 2024-03-12T03:21:49.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.361+0000 7fec1a7fc700 1 -- 172.21.15.118:0/3029969737 <== osd.5 v2:172.21.15.177:6816/3801270167 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fec24061430 con 0x7fec10072560 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] conn(0x7fec10072560 msgr2=0x7fec10074980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] conn(0x7fec10072560 0x7fec10074980 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fec1006c880 msgr2=0x7fec1006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fec1006c880 0x7fec1006ed40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec1c005950 tx=0x7fec1c00b410 comp rx=0 tx=0).stop 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24106a50 msgr2=0x7fec24194af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24106a50 0x7fec24194af0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fec1400bd70 tx=0x7fec1400bda0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 shutdown_connections 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fec1006c880 0x7fec1006ed40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:6816/3801270167,v1:172.21.15.177:6817/3801270167] conn(0x7fec10072560 0x7fec10074980 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fec24100a50 0x7fec241945b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 --2- 172.21.15.118:0/3029969737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fec24106a50 0x7fec24194af0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:49.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 >> 172.21.15.118:0/3029969737 conn(0x7fec240fc540 msgr2=0x7fec240fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:49.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 shutdown_connections 2024-03-12T03:21:49.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:49.362+0000 7fec2b649700 1 -- 172.21.15.118:0/3029969737 wait complete. 2024-03-12T03:21:49.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:49 smithi118 ceph-mon[29160]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:49.889 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:49 smithi177 ceph-mon[37140]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:50.817 INFO:teuthology.orchestra.run.smithi118.stdout:171798691848 2024-03-12T03:21:50.818 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 -- 172.21.15.118:0/1636758908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 msgr2=0x7f0e341047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 --2- 172.21.15.118:0/1636758908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e341047e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009910 tx=0x7f0e28009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 -- 172.21.15.118:0/1636758908 shutdown_connections 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 --2- 172.21.15.118:0/1636758908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e341047e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 --2- 172.21.15.118:0/1636758908 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e340ff9f0 0x7f0e340ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 -- 172.21.15.118:0/1636758908 >> 172.21.15.118:0/1636758908 conn(0x7f0e340fb450 msgr2=0x7f0e340fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 -- 172.21.15.118:0/1636758908 shutdown_connections 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 -- 172.21.15.118:0/1636758908 wait complete. 2024-03-12T03:21:51.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 Processor -- start 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.778+0000 7f0e39281700 1 -- start start 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e39281700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e340ff9f0 0x7f0e3419d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e39281700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e3419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e39281700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3419dc60 con 0x7f0e34100310 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e39281700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e341a1930 con 0x7f0e340ff9f0 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e327fc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e3419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e327fc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e3419d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37998/0 (socket says 172.21.15.118:37998) 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e327fc700 1 -- 172.21.15.118:0/3127715963 learned_addr learned my addr 172.21.15.118:0/3127715963 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e32ffd700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e340ff9f0 0x7f0e3419d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e327fc700 1 -- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e340ff9f0 msgr2=0x7f0e3419d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e327fc700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e340ff9f0 0x7f0e3419d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e327fc700 1 -- 172.21.15.118:0/3127715963 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e280095f0 con 0x7f0e34100310 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e327fc700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e3419d580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e280092e0 tx=0x7f0e2800bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2801d070 con 0x7f0e34100310 2024-03-12T03:21:51.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e28005700 con 0x7f0e34100310 2024-03-12T03:21:51.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e39281700 1 -- 172.21.15.118:0/3127715963 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e341a1bb0 con 0x7f0e34100310 2024-03-12T03:21:51.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2800fdb0 con 0x7f0e34100310 2024-03-12T03:21:51.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.779+0000 7f0e39281700 1 -- 172.21.15.118:0/3127715963 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e341a20a0 con 0x7f0e34100310 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.780+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0e341144d0 con 0x7f0e34100310 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.781+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f0e28017450 con 0x7f0e34100310 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.781+0000 7f0e1bfff700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f0e1c06c880 0x7f0e1c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.781+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f0e2808cc80 con 0x7f0e34100310 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.781+0000 7f0e32ffd700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f0e1c06c880 0x7f0e1c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.781+0000 7f0e1bfff700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] conn(0x7f0e1c072560 0x7f0e1c074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.781+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 --> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0e1c075030 con 0x7f0e1c072560 2024-03-12T03:21:51.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.781+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e28092360 con 0x7f0e34100310 2024-03-12T03:21:51.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.782+0000 7f0e32ffd700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f0e1c06c880 0x7f0e1c06ed40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e24007be0 tx=0x7f0e240073d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:51.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.782+0000 7f0e337fe700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] conn(0x7f0e1c072560 0x7f0e1c074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:51.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.782+0000 7f0e337fe700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] conn(0x7f0e1c072560 0x7f0e1c074980 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:51.786 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:51 smithi118 ceph-mon[29160]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:51.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.783+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== osd.3 v2:172.21.15.177:6800/2319016844 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0e1c075030 con 0x7f0e1c072560 2024-03-12T03:21:51.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.799+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 --> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0e341a25c0 con 0x7f0e1c072560 2024-03-12T03:21:51.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e1bfff700 1 -- 172.21.15.118:0/3127715963 <== osd.3 v2:172.21.15.177:6800/2319016844 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0e341a25c0 con 0x7f0e1c072560 2024-03-12T03:21:51.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] conn(0x7f0e1c072560 msgr2=0x7f0e1c074980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] conn(0x7f0e1c072560 0x7f0e1c074980 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f0e1c06c880 msgr2=0x7f0e1c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f0e1c06c880 0x7f0e1c06ed40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0e24007be0 tx=0x7f0e240073d0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 msgr2=0x7f0e3419d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e3419d580 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0e280092e0 tx=0x7f0e2800bd70 comp rx=0 tx=0).stop 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 shutdown_connections 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:6800/2319016844,v1:172.21.15.177:6801/2319016844] conn(0x7f0e1c072560 0x7f0e1c074980 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f0e1c06c880 0x7f0e1c06ed40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0e340ff9f0 0x7f0e3419d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 --2- 172.21.15.118:0/3127715963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0e34100310 0x7f0e3419d580 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 >> 172.21.15.118:0/3127715963 conn(0x7f0e340fb450 msgr2=0x7f0e3410e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 shutdown_connections 2024-03-12T03:21:51.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:51.800+0000 7f0e19ffb700 1 -- 172.21.15.118:0/3127715963 wait complete. 2024-03-12T03:21:51.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:51 smithi177 ceph-mon[37140]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 -- 172.21.15.118:0/753344583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 msgr2=0x7f759c102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 --2- 172.21.15.118:0/753344583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c102040 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f758c009940 tx=0x7f758c009c50 comp rx=0 tx=0).stop 2024-03-12T03:21:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 -- 172.21.15.118:0/753344583 shutdown_connections 2024-03-12T03:21:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 --2- 172.21.15.118:0/753344583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f759c102580 0x7f759c10aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 --2- 172.21.15.118:0/753344583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c102040 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 -- 172.21.15.118:0/753344583 >> 172.21.15.118:0/753344583 conn(0x7f759c0fb450 msgr2=0x7f759c0fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 -- 172.21.15.118:0/753344583 shutdown_connections 2024-03-12T03:21:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.473+0000 7f75a2383700 1 -- 172.21.15.118:0/753344583 wait complete. 2024-03-12T03:21:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f75a2383700 1 Processor -- start 2024-03-12T03:21:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f75a2383700 1 -- start start 2024-03-12T03:21:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f75a2383700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f75a2383700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f759c102580 0x7f759c19cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f75a2383700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c19d260 con 0x7f759c101c60 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f75a2383700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c196820 con 0x7f759c102580 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f759bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f759bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c19c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38014/0 (socket says 172.21.15.118:38014) 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f759bfff700 1 -- 172.21.15.118:0/3925527139 learned_addr learned my addr 172.21.15.118:0/3925527139 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f759b7fe700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f759c102580 0x7f759c19cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f759bfff700 1 -- 172.21.15.118:0/3925527139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f759c102580 msgr2=0x7f759c19cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f759bfff700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f759c102580 0x7f759c19cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.474+0000 7f759bfff700 1 -- 172.21.15.118:0/3925527139 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c0095f0 con 0x7f759c101c60 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.475+0000 7f759bfff700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c19c690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f758c005620 tx=0x7f758c005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.475+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758c01d070 con 0x7f759c101c60 2024-03-12T03:21:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.475+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f758c00f460 con 0x7f759c101c60 2024-03-12T03:21:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.475+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c196aa0 con 0x7f759c101c60 2024-03-12T03:21:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.475+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f758c021620 con 0x7f759c101c60 2024-03-12T03:21:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.475+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c196f90 con 0x7f759c101c60 2024-03-12T03:21:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.476+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f759c10df80 con 0x7f759c101c60 2024-03-12T03:21:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.476+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f758c00fab0 con 0x7f759c101c60 2024-03-12T03:21:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f75997fa700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f758406c880 0x7f758406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f758c08cde0 con 0x7f759c101c60 2024-03-12T03:21:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f759b7fe700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f758406c880 0x7f758406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f75997fa700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] conn(0x7f7584072560 0x7f7584074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 --> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7584075030 con 0x7f7584072560 2024-03-12T03:21:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f758c08d240 con 0x7f759c101c60 2024-03-12T03:21:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f75a0920700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] conn(0x7f7584072560 0x7f7584074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f759b7fe700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f758406c880 0x7f758406ed40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7590005950 tx=0x7f75900058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.477+0000 7f75a0920700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] conn(0x7f7584072560 0x7f7584074980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.478+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== osd.1 v2:172.21.15.118:6810/1159607125 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f7584075030 con 0x7f7584072560 2024-03-12T03:21:52.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.493+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 --> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f759c197440 con 0x7f7584072560 2024-03-12T03:21:52.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.493+0000 7f75997fa700 1 -- 172.21.15.118:0/3925527139 <== osd.1 v2:172.21.15.118:6810/1159607125 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f759c197440 con 0x7f7584072560 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] conn(0x7f7584072560 msgr2=0x7f7584074980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] conn(0x7f7584072560 0x7f7584074980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f758406c880 msgr2=0x7f758406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f758406c880 0x7f758406ed40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7590005950 tx=0x7f75900058e0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 msgr2=0x7f759c19c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c19c690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f758c005620 tx=0x7f758c005700 comp rx=0 tx=0).stop 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 shutdown_connections 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6810/1159607125,v1:172.21.15.118:6811/1159607125] conn(0x7f7584072560 0x7f7584074980 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f758406c880 0x7f758406ed40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f759c101c60 0x7f759c19c690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 --2- 172.21.15.118:0/3925527139 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f759c102580 0x7f759c19cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 >> 172.21.15.118:0/3925527139 conn(0x7f759c0fb450 msgr2=0x7f759c105350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 shutdown_connections 2024-03-12T03:21:52.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:52.494+0000 7f75a2383700 1 -- 172.21.15.118:0/3925527139 wait complete. 2024-03-12T03:21:52.845 INFO:teuthology.orchestra.run.smithi118.stdout:115964117012 2024-03-12T03:21:52.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-12T03:21:53.630 INFO:teuthology.orchestra.run.smithi118.stdout:60129542174 2024-03-12T03:21:53.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T03:21:53.706 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:53.708 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:53.710 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:53.892 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:53 smithi118 ceph-mon[29160]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:53.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:53 smithi177 ceph-mon[37140]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:54.597 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:55.001 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:54 smithi118 ceph-mon[29160]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:55.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:54 smithi177 ceph-mon[37140]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:56.379 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:21:57.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 -- 172.21.15.118:0/915758995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a8100330 msgr2=0x7f29a8104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:57.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 --2- 172.21.15.118:0/915758995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a8100330 0x7f29a8104800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2998009910 tx=0x7f2998009c20 comp rx=0 tx=0).stop 2024-03-12T03:21:57.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 -- 172.21.15.118:0/915758995 shutdown_connections 2024-03-12T03:21:57.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 --2- 172.21.15.118:0/915758995 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a8100330 0x7f29a8104800 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:57.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 --2- 172.21.15.118:0/915758995 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a80ff980 0x7f29a80ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:57.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 -- 172.21.15.118:0/915758995 >> 172.21.15.118:0/915758995 conn(0x7f29a80fb490 msgr2=0x7f29a80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:57.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 -- 172.21.15.118:0/915758995 shutdown_connections 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.526+0000 7f29afb21700 1 -- 172.21.15.118:0/915758995 wait complete. 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29afb21700 1 Processor -- start 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29afb21700 1 -- start start 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29afb21700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a80ff980 0x7f29a819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29afb21700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a8100330 0x7f29a819d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29afb21700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a819dc00 con 0x7f29a80ff980 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29afb21700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a81a18d0 con 0x7f29a8100330 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad8bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a80ff980 0x7f29a819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad8bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a80ff980 0x7f29a819cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42752/0 (socket says 172.21.15.118:42752) 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad8bd700 1 -- 172.21.15.118:0/388130975 learned_addr learned my addr 172.21.15.118:0/388130975 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad0bc700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a8100330 0x7f29a819d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad8bd700 1 -- 172.21.15.118:0/388130975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a8100330 msgr2=0x7f29a819d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad8bd700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a8100330 0x7f29a819d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:57.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad8bd700 1 -- 172.21.15.118:0/388130975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29980095f0 con 0x7f29a80ff980 2024-03-12T03:21:57.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad0bc700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a8100330 0x7f29a819d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:21:57.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f29ad8bd700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a80ff980 0x7f29a819cfe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e920 tx=0x7f29a400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:57.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.527+0000 7f299effd700 1 -- 172.21.15.118:0/388130975 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29a4009790 con 0x7f29a80ff980 2024-03-12T03:21:57.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.528+0000 7f299effd700 1 -- 172.21.15.118:0/388130975 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29a400cba0 con 0x7f29a80ff980 2024-03-12T03:21:57.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.528+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a81a1bb0 con 0x7f29a80ff980 2024-03-12T03:21:57.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.528+0000 7f299effd700 1 -- 172.21.15.118:0/388130975 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29a4018650 con 0x7f29a80ff980 2024-03-12T03:21:57.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.528+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a81a2100 con 0x7f29a80ff980 2024-03-12T03:21:57.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.528+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a8069850 con 0x7f29a80ff980 2024-03-12T03:21:57.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.529+0000 7f299effd700 1 -- 172.21.15.118:0/388130975 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f29a400cd10 con 0x7f29a80ff980 2024-03-12T03:21:57.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.529+0000 7f299effd700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f299406c880 0x7f299406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:21:57.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.529+0000 7f299effd700 1 -- 172.21.15.118:0/388130975 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f29a4014070 con 0x7f29a80ff980 2024-03-12T03:21:57.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.529+0000 7f29ad0bc700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f299406c880 0x7f299406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:21:57.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.530+0000 7f29ad0bc700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f299406c880 0x7f299406ed40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29980059d0 tx=0x7f2998005940 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:21:57.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.534+0000 7f299effd700 1 -- 172.21.15.118:0/388130975 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f29a4052960 con 0x7f29a80ff980 2024-03-12T03:21:57.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.734+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f29a81a2430 con 0x7f29a80ff980 2024-03-12T03:21:57.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.734+0000 7f299effd700 1 -- 172.21.15.118:0/388130975 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f29a405a0c0 con 0x7f29a80ff980 2024-03-12T03:21:57.737 INFO:teuthology.orchestra.run.smithi118.stdout:150323855375 2024-03-12T03:21:57.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f299406c880 msgr2=0x7f299406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:57.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f299406c880 0x7f299406ed40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29980059d0 tx=0x7f2998005940 comp rx=0 tx=0).stop 2024-03-12T03:21:57.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a80ff980 msgr2=0x7f29a819cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:21:57.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a80ff980 0x7f29a819cfe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f29a400e920 tx=0x7f29a400ece0 comp rx=0 tx=0).stop 2024-03-12T03:21:57.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 shutdown_connections 2024-03-12T03:21:57.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f299406c880 0x7f299406ed40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:57.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f29a80ff980 0x7f29a819cfe0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:57.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 --2- 172.21.15.118:0/388130975 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f29a8100330 0x7f29a819d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:21:57.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 >> 172.21.15.118:0/388130975 conn(0x7f29a80fb490 msgr2=0x7f29a810e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:21:57.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 shutdown_connections 2024-03-12T03:21:57.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:21:57.737+0000 7f29afb21700 1 -- 172.21.15.118:0/388130975 wait complete. 2024-03-12T03:21:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:57 smithi177 ceph-mon[37140]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:58.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:57 smithi118 ceph-mon[29160]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:58.742 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855373 got 150323855375 for osd.4 2024-03-12T03:21:58.742 DEBUG:teuthology.parallel:result is None 2024-03-12T03:21:58.878 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:58 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/388130975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:21:58.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:58 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/388130975' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:21:59.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:21:59 smithi118 ceph-mon[29160]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:21:59.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:21:59 smithi177 ceph-mon[37140]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:01.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 -- 172.21.15.118:0/3412860507 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 msgr2=0x7f7d680ff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:01.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 --2- 172.21.15.118:0/3412860507 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d680ff6c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7d54009910 tx=0x7f7d54009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 -- 172.21.15.118:0/3412860507 shutdown_connections 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 --2- 172.21.15.118:0/3412860507 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d680ff6c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 --2- 172.21.15.118:0/3412860507 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d68105a60 0x7f7d68105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 -- 172.21.15.118:0/3412860507 >> 172.21.15.118:0/3412860507 conn(0x7f7d68074c80 msgr2=0x7f7d68075090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 -- 172.21.15.118:0/3412860507 shutdown_connections 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.746+0000 7f7d6f192700 1 -- 172.21.15.118:0/3412860507 wait complete. 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6f192700 1 Processor -- start 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6f192700 1 -- start start 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6f192700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d6819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6f192700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d68105a60 0x7f7d6819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6f192700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6819f4e0 con 0x7f7d680ff260 2024-03-12T03:22:01.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6f192700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68198ac0 con 0x7f7d68105a60 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6cf2e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d6819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6cf2e700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d6819e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42780/0 (socket says 172.21.15.118:42780) 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d6cf2e700 1 -- 172.21.15.118:0/1571149478 learned_addr learned my addr 172.21.15.118:0/1571149478 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.747+0000 7f7d67fff700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d68105a60 0x7f7d6819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d6cf2e700 1 -- 172.21.15.118:0/1571149478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d68105a60 msgr2=0x7f7d6819eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d6cf2e700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d68105a60 0x7f7d6819eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d6cf2e700 1 -- 172.21.15.118:0/1571149478 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d540095f0 con 0x7f7d680ff260 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d67fff700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d68105a60 0x7f7d6819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d6cf2e700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d6819e980 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00b480 tx=0x7f7d5c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d65ffb700 1 -- 172.21.15.118:0/1571149478 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d5c0118c0 con 0x7f7d680ff260 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d68198da0 con 0x7f7d680ff260 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d65ffb700 1 -- 172.21.15.118:0/1571149478 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d5c010380 con 0x7f7d680ff260 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d65ffb700 1 -- 172.21.15.118:0/1571149478 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d5c00f4f0 con 0x7f7d680ff260 2024-03-12T03:22:01.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.748+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d68199370 con 0x7f7d680ff260 2024-03-12T03:22:01.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.749+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d68104e40 con 0x7f7d680ff260 2024-03-12T03:22:01.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.750+0000 7f7d65ffb700 1 -- 172.21.15.118:0/1571149478 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7d5c0104f0 con 0x7f7d680ff260 2024-03-12T03:22:01.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.750+0000 7f7d65ffb700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d5806c880 0x7f7d5806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:01.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.750+0000 7f7d65ffb700 1 -- 172.21.15.118:0/1571149478 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d5c08b500 con 0x7f7d680ff260 2024-03-12T03:22:01.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.750+0000 7f7d67fff700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d5806c880 0x7f7d5806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:01.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.750+0000 7f7d67fff700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d5806c880 0x7f7d5806ed40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d68199c80 tx=0x7f7d5401a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:01.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.755+0000 7f7d65ffb700 1 -- 172.21.15.118:0/1571149478 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7d5c059960 con 0x7f7d680ff260 2024-03-12T03:22:01.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:01 smithi118 ceph-mon[29160]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:01.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:01 smithi177 ceph-mon[37140]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:01.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.962+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7d68068bc0 con 0x7f7d680ff260 2024-03-12T03:22:01.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.962+0000 7f7d65ffb700 1 -- 172.21.15.118:0/1571149478 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7d5c0594f0 con 0x7f7d680ff260 2024-03-12T03:22:01.964 INFO:teuthology.orchestra.run.smithi118.stdout:171798691850 2024-03-12T03:22:01.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d5806c880 msgr2=0x7f7d5806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:01.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d5806c880 0x7f7d5806ed40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d68199c80 tx=0x7f7d5401a040 comp rx=0 tx=0).stop 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 msgr2=0x7f7d6819e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d6819e980 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00b480 tx=0x7f7d5c00b840 comp rx=0 tx=0).stop 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 shutdown_connections 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d5806c880 0x7f7d5806ed40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d680ff260 0x7f7d6819e980 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 --2- 172.21.15.118:0/1571149478 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7d68105a60 0x7f7d6819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 >> 172.21.15.118:0/1571149478 conn(0x7f7d68074c80 msgr2=0x7f7d680fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 shutdown_connections 2024-03-12T03:22:01.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:01.964+0000 7f7d6f192700 1 -- 172.21.15.118:0/1571149478 wait complete. 2024-03-12T03:22:02.716 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691848 got 171798691850 for osd.5 2024-03-12T03:22:02.717 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:02.900 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:02 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1571149478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:22:02.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:02 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1571149478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:22:03.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 -- 172.21.15.118:0/2064199786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 msgr2=0x7f0188100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 --2- 172.21.15.118:0/2064199786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f0188100ed0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0178009910 tx=0x7f0178009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:03.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 -- 172.21.15.118:0/2064199786 shutdown_connections 2024-03-12T03:22:03.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 --2- 172.21.15.118:0/2064199786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f0188100ed0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 --2- 172.21.15.118:0/2064199786 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0188106a50 0x7f0188106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 -- 172.21.15.118:0/2064199786 >> 172.21.15.118:0/2064199786 conn(0x7f01880fc540 msgr2=0x7f01880fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 -- 172.21.15.118:0/2064199786 shutdown_connections 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 -- 172.21.15.118:0/2064199786 wait complete. 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 Processor -- start 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.099+0000 7f01905b7700 1 -- start start 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f01905b7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f018819c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f01905b7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0188106a50 0x7f018819cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f01905b7700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018819d1f0 con 0x7f0188100a50 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f01905b7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01881967f0 con 0x7f0188106a50 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018e353700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f018819c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018e353700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f018819c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42786/0 (socket says 172.21.15.118:42786) 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018e353700 1 -- 172.21.15.118:0/2236322493 learned_addr learned my addr 172.21.15.118:0/2236322493 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018db52700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0188106a50 0x7f018819cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018e353700 1 -- 172.21.15.118:0/2236322493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0188106a50 msgr2=0x7f018819cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018e353700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0188106a50 0x7f018819cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018e353700 1 -- 172.21.15.118:0/2236322493 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01780095f0 con 0x7f0188100a50 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018db52700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0188106a50 0x7f018819cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f018e353700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f018819c6b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f018400c930 tx=0x7f018400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f017f7fe700 1 -- 172.21.15.118:0/2236322493 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0184007ad0 con 0x7f0188100a50 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f017f7fe700 1 -- 172.21.15.118:0/2236322493 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0184007c30 con 0x7f0188100a50 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f017f7fe700 1 -- 172.21.15.118:0/2236322493 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0184018610 con 0x7f0188100a50 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0188196ad0 con 0x7f0188100a50 2024-03-12T03:22:03.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.100+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0188196fd0 con 0x7f0188100a50 2024-03-12T03:22:03.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.101+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0188109070 con 0x7f0188100a50 2024-03-12T03:22:03.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.102+0000 7f017f7fe700 1 -- 172.21.15.118:0/2236322493 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f018400f450 con 0x7f0188100a50 2024-03-12T03:22:03.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.102+0000 7f017f7fe700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f017406c880 0x7f017406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:03.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.103+0000 7f017f7fe700 1 -- 172.21.15.118:0/2236322493 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f018408bb70 con 0x7f0188100a50 2024-03-12T03:22:03.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.103+0000 7f018db52700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f017406c880 0x7f017406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:03.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.103+0000 7f018db52700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f017406c880 0x7f017406ed40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0188197c10 tx=0x7f017801a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:03.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.107+0000 7f017f7fe700 1 -- 172.21.15.118:0/2236322493 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f0184059f50 con 0x7f0188100a50 2024-03-12T03:22:03.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.305+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0188197d50 con 0x7f0188100a50 2024-03-12T03:22:03.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.306+0000 7f017f7fe700 1 -- 172.21.15.118:0/2236322493 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0184059ae0 con 0x7f0188100a50 2024-03-12T03:22:03.308 INFO:teuthology.orchestra.run.smithi118.stdout:38654705702 2024-03-12T03:22:03.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f017406c880 msgr2=0x7f017406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f017406c880 0x7f017406ed40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0188197c10 tx=0x7f017801a040 comp rx=0 tx=0).stop 2024-03-12T03:22:03.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 msgr2=0x7f018819c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f018819c6b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f018400c930 tx=0x7f018400ccf0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 shutdown_connections 2024-03-12T03:22:03.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f017406c880 0x7f017406ed40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0188100a50 0x7f018819c6b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 --2- 172.21.15.118:0/2236322493 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f0188106a50 0x7f018819cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 >> 172.21.15.118:0/2236322493 conn(0x7f01880fc540 msgr2=0x7f01880fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:03.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 shutdown_connections 2024-03-12T03:22:03.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.308+0000 7f01905b7700 1 -- 172.21.15.118:0/2236322493 wait complete. 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 -- 172.21.15.118:0/1033008673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 msgr2=0x7f9e1c10aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 --2- 172.21.15.118:0/1033008673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c10aaa0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c009910 tx=0x7f9e0c009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 -- 172.21.15.118:0/1033008673 shutdown_connections 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 --2- 172.21.15.118:0/1033008673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c10aaa0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 --2- 172.21.15.118:0/1033008673 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e1c101c90 0x7f9e1c102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 -- 172.21.15.118:0/1033008673 >> 172.21.15.118:0/1033008673 conn(0x7f9e1c0fb440 msgr2=0x7f9e1c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 -- 172.21.15.118:0/1033008673 shutdown_connections 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 -- 172.21.15.118:0/1033008673 wait complete. 2024-03-12T03:22:03.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.324+0000 7f9e23c04700 1 Processor -- start 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e23c04700 1 -- start start 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e23c04700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e1c101c90 0x7f9e1c0ff8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e23c04700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c0ffe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e23c04700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c100370 con 0x7f9e1c1025b0 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e23c04700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c1004e0 con 0x7f9e1c101c90 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e2119f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c0ffe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e2119f700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c0ffe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42808/0 (socket says 172.21.15.118:42808) 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e2119f700 1 -- 172.21.15.118:0/3562238984 learned_addr learned my addr 172.21.15.118:0/3562238984 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e219a0700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e1c101c90 0x7f9e1c0ff8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e2119f700 1 -- 172.21.15.118:0/3562238984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e1c101c90 msgr2=0x7f9e1c0ff8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e2119f700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e1c101c90 0x7f9e1c0ff8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e2119f700 1 -- 172.21.15.118:0/3562238984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c0095f0 con 0x7f9e1c1025b0 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.325+0000 7f9e2119f700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c0ffe30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00bd70 tx=0x7f9e0c00bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.326+0000 7f9e12ffd700 1 -- 172.21.15.118:0/3562238984 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0c01d070 con 0x7f9e1c1025b0 2024-03-12T03:22:03.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.326+0000 7f9e12ffd700 1 -- 172.21.15.118:0/3562238984 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e0c0056c0 con 0x7f9e1c1025b0 2024-03-12T03:22:03.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.326+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c100760 con 0x7f9e1c1025b0 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.326+0000 7f9e12ffd700 1 -- 172.21.15.118:0/3562238984 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0c017400 con 0x7f9e1c1025b0 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.326+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c1a2e30 con 0x7f9e1c1025b0 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.327+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c0694d0 con 0x7f9e1c1025b0 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.328+0000 7f9e12ffd700 1 -- 172.21.15.118:0/3562238984 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9e0c003a60 con 0x7f9e1c1025b0 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.328+0000 7f9e12ffd700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9e0806c880 0x7f9e0806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.328+0000 7f9e12ffd700 1 -- 172.21.15.118:0/3562238984 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f9e0c093410 con 0x7f9e1c1025b0 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.328+0000 7f9e219a0700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9e0806c880 0x7f9e0806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:03.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.329+0000 7f9e219a0700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9e0806c880 0x7f9e0806ed40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9e18005950 tx=0x7f9e1800a300 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:03.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.333+0000 7f9e12ffd700 1 -- 172.21.15.118:0/3562238984 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9e0c0c3960 con 0x7f9e1c1025b0 2024-03-12T03:22:03.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.534+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9e1c100760 con 0x7f9e1c1025b0 2024-03-12T03:22:03.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.534+0000 7f9e12ffd700 1 -- 172.21.15.118:0/3562238984 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9e1c100760 con 0x7f9e1c1025b0 2024-03-12T03:22:03.537 INFO:teuthology.orchestra.run.smithi118.stdout:60129542176 2024-03-12T03:22:03.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9e0806c880 msgr2=0x7f9e0806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9e0806c880 0x7f9e0806ed40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9e18005950 tx=0x7f9e1800a300 comp rx=0 tx=0).stop 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 msgr2=0x7f9e1c0ffe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c0ffe30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9e0c00bd70 tx=0x7f9e0c00bda0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 shutdown_connections 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9e0806c880 0x7f9e0806ed40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e1c101c90 0x7f9e1c0ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 --2- 172.21.15.118:0/3562238984 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e1c1025b0 0x7f9e1c0ffe30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 >> 172.21.15.118:0/3562238984 conn(0x7f9e1c0fb440 msgr2=0x7f9e1c102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 shutdown_connections 2024-03-12T03:22:03.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:03.537+0000 7f9e23c04700 1 -- 172.21.15.118:0/3562238984 wait complete. 2024-03-12T03:22:03.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:03 smithi118 ceph-mon[29160]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:03.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:03 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2236322493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:22:03.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:03 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3562238984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:22:03.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:03.944 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705698 got 38654705702 for osd.0 2024-03-12T03:22:03.945 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:03.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:03 smithi177 ceph-mon[37140]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:03.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:03 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2236322493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:22:03.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:03 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3562238984' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:22:03.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:04.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 -- 172.21.15.118:0/3036830968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c0ff7e0 msgr2=0x7f3e8c0ffbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:04.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3036830968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c0ff7e0 0x7f3e8c0ffbc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400b0a0 tx=0x7f3e8400b3b0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 -- 172.21.15.118:0/3036830968 shutdown_connections 2024-03-12T03:22:04.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3036830968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e8c100190 0x7f3e8c10ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3036830968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c0ff7e0 0x7f3e8c0ffbc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 -- 172.21.15.118:0/3036830968 >> 172.21.15.118:0/3036830968 conn(0x7f3e8c0765c0 msgr2=0x7f3e8c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 -- 172.21.15.118:0/3036830968 shutdown_connections 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.543+0000 7f3e92abd700 1 -- 172.21.15.118:0/3036830968 wait complete. 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e92abd700 1 Processor -- start 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e92abd700 1 -- start start 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e92abd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e8c0ff7e0 0x7f3e8c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e92abd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c100190 0x7f3e8c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e92abd700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c19f550 con 0x7f3e8c100190 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e92abd700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c198b30 con 0x7f3e8c0ff7e0 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c100190 0x7f3e8c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e8bfff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c100190 0x7f3e8c19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48244/0 (socket says 172.21.15.118:48244) 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e8bfff700 1 -- 172.21.15.118:0/3684729968 learned_addr learned my addr 172.21.15.118:0/3684729968 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e8bfff700 1 -- 172.21.15.118:0/3684729968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e8c0ff7e0 msgr2=0x7f3e8c19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e8bfff700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e8c0ff7e0 0x7f3e8c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.544+0000 7f3e8bfff700 1 -- 172.21.15.118:0/3684729968 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84009d00 con 0x7f3e8c100190 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.545+0000 7f3e8bfff700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c100190 0x7f3e8c19ef30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b4b0 tx=0x7f3e7c00b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.545+0000 7f3e89ffb700 1 -- 172.21.15.118:0/3684729968 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7c011800 con 0x7f3e8c100190 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.545+0000 7f3e89ffb700 1 -- 172.21.15.118:0/3684729968 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e7c011e20 con 0x7f3e8c100190 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.545+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c198e10 con 0x7f3e8c100190 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.545+0000 7f3e89ffb700 1 -- 172.21.15.118:0/3684729968 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e7c00f2e0 con 0x7f3e8c100190 2024-03-12T03:22:04.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.545+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c199360 con 0x7f3e8c100190 2024-03-12T03:22:04.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.546+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c107830 con 0x7f3e8c100190 2024-03-12T03:22:04.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.546+0000 7f3e89ffb700 1 -- 172.21.15.118:0/3684729968 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f3e7c010380 con 0x7f3e8c100190 2024-03-12T03:22:04.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.546+0000 7f3e89ffb700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f3e7806c880 0x7f3e7806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:04.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.547+0000 7f3e90859700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f3e7806c880 0x7f3e7806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:04.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.547+0000 7f3e89ffb700 1 -- 172.21.15.118:0/3684729968 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f3e7c08b510 con 0x7f3e8c100190 2024-03-12T03:22:04.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.547+0000 7f3e90859700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f3e7806c880 0x7f3e7806ed40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400b830 tx=0x7f3e840092e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:04.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.551+0000 7f3e89ffb700 1 -- 172.21.15.118:0/3684729968 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f3e7c0598f0 con 0x7f3e8c100190 2024-03-12T03:22:04.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.750+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f3e8c068c60 con 0x7f3e8c100190 2024-03-12T03:22:04.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.750+0000 7f3e89ffb700 1 -- 172.21.15.118:0/3684729968 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3e7c059480 con 0x7f3e8c100190 2024-03-12T03:22:04.752 INFO:teuthology.orchestra.run.smithi118.stdout:81604378651 2024-03-12T03:22:04.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f3e7806c880 msgr2=0x7f3e7806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:04.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f3e7806c880 0x7f3e7806ed40 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400b830 tx=0x7f3e840092e0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c100190 msgr2=0x7f3e8c19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:04.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c100190 0x7f3e8c19ef30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c00b4b0 tx=0x7f3e7c00b870 comp rx=0 tx=0).stop 2024-03-12T03:22:04.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 shutdown_connections 2024-03-12T03:22:04.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f3e7806c880 0x7f3e7806ed40 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f3e8c0ff7e0 0x7f3e8c19e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 --2- 172.21.15.118:0/3684729968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e8c100190 0x7f3e8c19ef30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:04.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 >> 172.21.15.118:0/3684729968 conn(0x7f3e8c0765c0 msgr2=0x7f3e8c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:04.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 shutdown_connections 2024-03-12T03:22:04.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:04.752+0000 7f3e92abd700 1 -- 172.21.15.118:0/3684729968 wait complete. 2024-03-12T03:22:05.011 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:04 smithi118 ceph-mon[29160]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:05.012 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:04 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3684729968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:22:05.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:04 smithi177 ceph-mon[37140]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:05.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:04 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3684729968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:22:05.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 -- 172.21.15.118:0/3355655518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 msgr2=0x7fd6c8101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:05.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 --2- 172.21.15.118:0/3355655518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c8101fb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8009910 tx=0x7fd6b8009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:05.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 -- 172.21.15.118:0/3355655518 shutdown_connections 2024-03-12T03:22:05.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 --2- 172.21.15.118:0/3355655518 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6c81024f0 0x7fd6c810a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:05.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 --2- 172.21.15.118:0/3355655518 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c8101fb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:05.383 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 -- 172.21.15.118:0/3355655518 >> 172.21.15.118:0/3355655518 conn(0x7fd6c80765c0 msgr2=0x7fd6c80fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:05.383 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 -- 172.21.15.118:0/3355655518 shutdown_connections 2024-03-12T03:22:05.383 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.379+0000 7fd6ce542700 1 -- 172.21.15.118:0/3355655518 wait complete. 2024-03-12T03:22:05.383 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6ce542700 1 Processor -- start 2024-03-12T03:22:05.383 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6ce542700 1 -- start start 2024-03-12T03:22:05.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6ce542700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:05.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6ce542700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6c81024f0 0x7fd6c819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:05.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6ce542700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c819d2f0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6ce542700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c8196870 con 0x7fd6c81024f0 2024-03-12T03:22:05.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6c7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:05.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6c7fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c819c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48258/0 (socket says 172.21.15.118:48258) 2024-03-12T03:22:05.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6c7fff700 1 -- 172.21.15.118:0/1116372143 learned_addr learned my addr 172.21.15.118:0/1116372143 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.380+0000 7fd6c77fe700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6c81024f0 0x7fd6c819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c7fff700 1 -- 172.21.15.118:0/1116372143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6c81024f0 msgr2=0x7fd6c819cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c7fff700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6c81024f0 0x7fd6c819cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c7fff700 1 -- 172.21.15.118:0/1116372143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80095f0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c77fe700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6c81024f0 0x7fd6c819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c7fff700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c819c6d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8005f10 tx=0x7fd6b800b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c57fa700 1 -- 172.21.15.118:0/1116372143 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6b801d070 con 0x7fd6c8101bd0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c57fa700 1 -- 172.21.15.118:0/1116372143 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd6b80052a0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c8196af0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6c57fa700 1 -- 172.21.15.118:0/1116372143 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd6b8022470 con 0x7fd6c8101bd0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.381+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c8196fe0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.382+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c8068ce0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.383+0000 7fd6c57fa700 1 -- 172.21.15.118:0/1116372143 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fd6b8003780 con 0x7fd6c8101bd0 2024-03-12T03:22:05.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.383+0000 7fd6c57fa700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd6b006c880 0x7fd6b006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:05.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.383+0000 7fd6c77fe700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd6b006c880 0x7fd6b006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:05.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.383+0000 7fd6c57fa700 1 -- 172.21.15.118:0/1116372143 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fd6b808cd00 con 0x7fd6c8101bd0 2024-03-12T03:22:05.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.383+0000 7fd6c77fe700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd6b006c880 0x7fd6b006ed40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8197fa0 tx=0x7fd6bc009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:05.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.388+0000 7fd6c57fa700 1 -- 172.21.15.118:0/1116372143 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fd6b8090030 con 0x7fd6c8101bd0 2024-03-12T03:22:05.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.587+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd6c80623e0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.587+0000 7fd6c57fa700 1 -- 172.21.15.118:0/1116372143 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd6c80623e0 con 0x7fd6c8101bd0 2024-03-12T03:22:05.590 INFO:teuthology.orchestra.run.smithi118.stdout:115964117015 2024-03-12T03:22:05.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd6b006c880 msgr2=0x7fd6b006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:05.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd6b006c880 0x7fd6b006ed40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8197fa0 tx=0x7fd6bc009260 comp rx=0 tx=0).stop 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 msgr2=0x7fd6c819c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c819c6d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8005f10 tx=0x7fd6b800b650 comp rx=0 tx=0).stop 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 shutdown_connections 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fd6b006c880 0x7fd6b006ed40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6c8101bd0 0x7fd6c819c6d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 --2- 172.21.15.118:0/1116372143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fd6c81024f0 0x7fd6c819cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 >> 172.21.15.118:0/1116372143 conn(0x7fd6c80765c0 msgr2=0x7fd6c8102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 shutdown_connections 2024-03-12T03:22:05.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:05.589+0000 7fd6ce542700 1 -- 172.21.15.118:0/1116372143 wait complete. 2024-03-12T03:22:05.875 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542176 for osd.1 2024-03-12T03:22:05.876 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:06.189 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:05 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1116372143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:22:06.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:05 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1116372143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:22:06.404 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378651 for osd.2 2024-03-12T03:22:06.404 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:07.036 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:06 smithi118 ceph-mon[29160]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:07.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117015 for osd.3 2024-03-12T03:22:07.060 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:07.060 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T03:22:07.060 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:22:07.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:06 smithi177 ceph-mon[37140]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:07.266 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 -- 172.21.15.118:0/1290594287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 msgr2=0x7f555410aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 --2- 172.21.15.118:0/1290594287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555410aa50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5548009910 tx=0x7f5548009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 -- 172.21.15.118:0/1290594287 shutdown_connections 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 --2- 172.21.15.118:0/1290594287 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555410aa50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 --2- 172.21.15.118:0/1290594287 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5554101c40 0x7f5554102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 -- 172.21.15.118:0/1290594287 >> 172.21.15.118:0/1290594287 conn(0x7f55540fb490 msgr2=0x7f55540fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 -- 172.21.15.118:0/1290594287 shutdown_connections 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 -- 172.21.15.118:0/1290594287 wait complete. 2024-03-12T03:22:08.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 Processor -- start 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.923+0000 7f555a70d700 1 -- start start 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f555a70d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5554101c40 0x7f555419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f555a70d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f555a70d700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f555419d380 con 0x7f5554102560 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f555a70d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5554196960 con 0x7f5554101c40 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55537fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55537fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48270/0 (socket says 172.21.15.118:48270) 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55537fe700 1 -- 172.21.15.118:0/3736719565 learned_addr learned my addr 172.21.15.118:0/3736719565 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f5553fff700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5554101c40 0x7f555419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55537fe700 1 -- 172.21.15.118:0/3736719565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5554101c40 msgr2=0x7f555419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55537fe700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5554101c40 0x7f555419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55537fe700 1 -- 172.21.15.118:0/3736719565 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55480095f0 con 0x7f5554102560 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55537fe700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555419cd60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f554800bd70 tx=0x7f554800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:08.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55517fa700 1 -- 172.21.15.118:0/3736719565 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f554801d070 con 0x7f5554102560 2024-03-12T03:22:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55517fa700 1 -- 172.21.15.118:0/3736719565 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f55480056c0 con 0x7f5554102560 2024-03-12T03:22:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5554196be0 con 0x7f5554102560 2024-03-12T03:22:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f55517fa700 1 -- 172.21.15.118:0/3736719565 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5548017400 con 0x7f5554102560 2024-03-12T03:22:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.924+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55541970d0 con 0x7f5554102560 2024-03-12T03:22:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.925+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55541053c0 con 0x7f5554102560 2024-03-12T03:22:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.926+0000 7f55517fa700 1 -- 172.21.15.118:0/3736719565 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5548003a60 con 0x7f5554102560 2024-03-12T03:22:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.926+0000 7f55517fa700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f553c06c880 0x7f553c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.927+0000 7f55517fa700 1 -- 172.21.15.118:0/3736719565 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f554808ce70 con 0x7f5554102560 2024-03-12T03:22:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.927+0000 7f5553fff700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f553c06c880 0x7f553c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.927+0000 7f5553fff700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f553c06c880 0x7f553c06ed40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5544009a10 tx=0x7f5544009190 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:08.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:08.930+0000 7f55517fa700 1 -- 172.21.15.118:0/3736719565 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f554805b370 con 0x7f5554102560 2024-03-12T03:22:09.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.121+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5554197e90 con 0x7f553c06c880 2024-03-12T03:22:09.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.122+0000 7f55517fa700 1 -- 172.21.15.118:0/3736719565 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28052 (secure 0 0 0) 0x7f5554197e90 con 0x7f553c06c880 2024-03-12T03:22:09.126 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:09.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f553c06c880 msgr2=0x7f553c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:09.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f553c06c880 0x7f553c06ed40 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5544009a10 tx=0x7f5544009190 comp rx=0 tx=0).stop 2024-03-12T03:22:09.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 msgr2=0x7f555419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:09.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555419cd60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f554800bd70 tx=0x7f554800bda0 comp rx=0 tx=0).stop 2024-03-12T03:22:09.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 shutdown_connections 2024-03-12T03:22:09.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f553c06c880 0x7f553c06ed40 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:09.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5554101c40 0x7f555419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:09.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 --2- 172.21.15.118:0/3736719565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5554102560 0x7f555419cd60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:09.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 >> 172.21.15.118:0/3736719565 conn(0x7f55540fb490 msgr2=0x7f5554102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:09.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 shutdown_connections 2024-03-12T03:22:09.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:09.125+0000 7f555a70d700 1 -- 172.21.15.118:0/3736719565 wait complete. 2024-03-12T03:22:09.129 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-12T03:22:09.735 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:09 smithi118 ceph-mon[29160]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:09.757 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-12T03:22:08.564085+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":166048,"kb_used_data":3920,"kb_used_omap":10,"kb_used_meta":162037,"kb_avail":562304864,"statfs":{"total":575970213888,"available":575800180736,"internally_reserved":0,"allocated":4014080,"data_stored":2627111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10686,"internal_metadata":165926466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002610"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-12T03:21:19.619311+0000","last_change":"2024-03-12T03:20:53.508743+0000","last_active":"2024-03-12T03:21:19.619311+0000","last_peered":"2024-03-12T03:21:19.619311+0000","last_clean":"2024-03-12T03:21:19.619311+0000","last_became_active":"2024-03-12T03:20:53.508426+0000","last_became_peered":"2024-03-12T03:20:53.508426+0000","last_unstale":"2024-03-12T03:21:19.619311+0000","last_undegraded":"2024-03-12T03:21:19.619311+0000","last_fullsized":"2024-03-12T03:21:19.619311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:20:05.167689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:20:05.167689+0000","last_clean_scrub_stamp":"2024-03-12T03:20:05.167689+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:08:30.983737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.438}]},{"osd":3,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.378},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":732760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58999999999999997}]},{"osd":1,"last update":"Tue Mar 12 03:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.623},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.60599999999999998}]},{"osd":2,"last update":"Tue Mar 12 03:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57899999999999996}]},{"osd":4,"last update":"Tue Mar 12 03:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.311}]},{"osd":1,"last update":"Tue Mar 12 03:21:04 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Mar 12 03:21:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.501},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Tue Mar 12 03:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.436},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.46400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 12 03:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.30599999999999999,"15min":0.30199999999999999},"min":{"1min":0.23200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.309,"15min":0.30299999999999999},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Tue Mar 12 03:21:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.40600000000000003}]},{"osd":3,"last update":"Tue Mar 12 03:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.629},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001}]},{"osd":4,"last update":"Tue Mar 12 03:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.30399999999999999,"15min":0.29799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.316,"15min":0.31},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.36799999999999999}]},{"osd":2,"last update":"Tue Mar 12 03:21:05 2024","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Mar 12 03:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.69099999999999995}]},{"osd":4,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.64500000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:22:09.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:22:09.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:09 smithi177 ceph-mon[37140]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:09.966 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:10.698 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:10 smithi118 ceph-mon[29160]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:22:10.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:10 smithi177 ceph-mon[37140]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:22:11.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 -- 172.21.15.118:0/3170855936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 msgr2=0x7f1c900ffdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:11.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3170855936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c900ffdc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1c80009910 tx=0x7f1c80009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:11.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 -- 172.21.15.118:0/3170855936 shutdown_connections 2024-03-12T03:22:11.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3170855936 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c900ffdc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:11.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3170855936 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c90105940 0x7f1c90105d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:11.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 -- 172.21.15.118:0/3170855936 >> 172.21.15.118:0/3170855936 conn(0x7f1c900fb450 msgr2=0x7f1c900fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:11.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 -- 172.21.15.118:0/3170855936 shutdown_connections 2024-03-12T03:22:11.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.552+0000 7f1c94f79700 1 -- 172.21.15.118:0/3170855936 wait complete. 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c94f79700 1 Processor -- start 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c94f79700 1 -- start start 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c94f79700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c9019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c94f79700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c90105940 0x7f1c9019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c94f79700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9019d2f0 con 0x7f1c900ff940 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c94f79700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c901968d0 con 0x7f1c90105940 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8e59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c9019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:11.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8e59c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c9019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48296/0 (socket says 172.21.15.118:48296) 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8e59c700 1 -- 172.21.15.118:0/3959219219 learned_addr learned my addr 172.21.15.118:0/3959219219 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8dd9b700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c90105940 0x7f1c9019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8e59c700 1 -- 172.21.15.118:0/3959219219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c90105940 msgr2=0x7f1c9019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8e59c700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c90105940 0x7f1c9019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8e59c700 1 -- 172.21.15.118:0/3959219219 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800095f0 con 0x7f1c900ff940 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.553+0000 7f1c8e59c700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c9019c790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400b4b0 tx=0x7f1c8400b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.554+0000 7f1c7f7fe700 1 -- 172.21.15.118:0/3959219219 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c84011800 con 0x7f1c900ff940 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.554+0000 7f1c7f7fe700 1 -- 172.21.15.118:0/3959219219 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1c84011e20 con 0x7f1c900ff940 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.554+0000 7f1c7f7fe700 1 -- 172.21.15.118:0/3959219219 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1c8400f2e0 con 0x7f1c900ff940 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.554+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c90196bb0 con 0x7f1c900ff940 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.554+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c901970b0 con 0x7f1c900ff940 2024-03-12T03:22:11.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.555+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c90108f70 con 0x7f1c900ff940 2024-03-12T03:22:11.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.555+0000 7f1c7f7fe700 1 -- 172.21.15.118:0/3959219219 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1c84010380 con 0x7f1c900ff940 2024-03-12T03:22:11.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.555+0000 7f1c7f7fe700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1c7806c880 0x7f1c7806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:11.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.556+0000 7f1c8dd9b700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1c7806c880 0x7f1c7806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:11.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.556+0000 7f1c7f7fe700 1 -- 172.21.15.118:0/3959219219 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1c8408b520 con 0x7f1c900ff940 2024-03-12T03:22:11.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.556+0000 7f1c8dd9b700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1c7806c880 0x7f1c7806ed40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c800059d0 tx=0x7f1c8001a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:11.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.560+0000 7f1c7f7fe700 1 -- 172.21.15.118:0/3959219219 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1c84059900 con 0x7f1c900ff940 2024-03-12T03:22:11.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.748+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c90197d90 con 0x7f1c7806c880 2024-03-12T03:22:11.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.750+0000 7f1c7f7fe700 1 -- 172.21.15.118:0/3959219219 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27954 (secure 0 0 0) 0x7f1c90197d90 con 0x7f1c7806c880 2024-03-12T03:22:11.752 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1c7806c880 msgr2=0x7f1c7806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1c7806c880 0x7f1c7806ed40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1c800059d0 tx=0x7f1c8001a040 comp rx=0 tx=0).stop 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 msgr2=0x7f1c9019c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c9019c790 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400b4b0 tx=0x7f1c8400b870 comp rx=0 tx=0).stop 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 shutdown_connections 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1c7806c880 0x7f1c7806ed40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1c900ff940 0x7f1c9019c790 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 --2- 172.21.15.118:0/3959219219 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1c90105940 0x7f1c9019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:11.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 >> 172.21.15.118:0/3959219219 conn(0x7f1c900fb450 msgr2=0x7f1c90104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:11.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 shutdown_connections 2024-03-12T03:22:11.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:11.752+0000 7f1c94f79700 1 -- 172.21.15.118:0/3959219219 wait complete. 2024-03-12T03:22:11.756 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-12T03:22:11.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:11 smithi118 ceph-mon[29160]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:11.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:11 smithi177 ceph-mon[37140]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:12.330 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-03-12T03:22:10.564472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":166048,"kb_used_data":3920,"kb_used_omap":10,"kb_used_meta":162037,"kb_avail":562304864,"statfs":{"total":575970213888,"available":575800180736,"internally_reserved":0,"allocated":4014080,"data_stored":2627111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10686,"internal_metadata":165926466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002569"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":185,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-12T03:21:19.619311+0000","last_change":"2024-03-12T03:20:53.508743+0000","last_active":"2024-03-12T03:21:19.619311+0000","last_peered":"2024-03-12T03:21:19.619311+0000","last_clean":"2024-03-12T03:21:19.619311+0000","last_became_active":"2024-03-12T03:20:53.508426+0000","last_became_peered":"2024-03-12T03:20:53.508426+0000","last_unstale":"2024-03-12T03:21:19.619311+0000","last_undegraded":"2024-03-12T03:21:19.619311+0000","last_fullsized":"2024-03-12T03:21:19.619311+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:20:05.167689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:20:05.167689+0000","last_clean_scrub_stamp":"2024-03-12T03:20:05.167689+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T14:08:30.983737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.49199999999999999}]},{"osd":1,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.46200000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.438}]},{"osd":3,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.378},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":962560,"data_stored":732760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58999999999999997}]},{"osd":1,"last update":"Tue Mar 12 03:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.623},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.60599999999999998}]},{"osd":2,"last update":"Tue Mar 12 03:21:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57899999999999996}]},{"osd":4,"last update":"Tue Mar 12 03:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27384,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717768,"statfs":{"total":95995035648,"available":95966994432,"internally_reserved":0,"allocated":372736,"data_stored":142723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.34300000000000003},"min":{"1min":0.249,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.47299999999999998,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36399999999999999,"15min":0.36499999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"last":0.39600000000000002}]},{"osd":1,"last update":"Tue Mar 12 03:22:07 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36499999999999999,"15min":0.36899999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.42899999999999999,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.377},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.38900000000000001,"15min":0.39300000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"last":0.41199999999999998}]},{"osd":3,"last update":"Tue Mar 12 03:21:32 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.67800000000000005}]},{"osd":4,"last update":"Tue Mar 12 03:21:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.624}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 12 03:21:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.30599999999999999,"15min":0.30199999999999999},"min":{"1min":0.23200000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.309,"15min":0.30299999999999999},"min":{"1min":0.255,"5min":0.216,"15min":0.216},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Tue Mar 12 03:21:09 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.40600000000000003}]},{"osd":3,"last update":"Tue Mar 12 03:21:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.629},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001}]},{"osd":4,"last update":"Tue Mar 12 03:21:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.58199999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":733091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:21:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.30399999999999999,"15min":0.29799999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.316,"15min":0.31},"min":{"1min":0.222,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:22:08 2024","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.33200000000000002,"15min":0.33400000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.29199999999999998}]},{"osd":3,"last update":"Tue Mar 12 03:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.51700000000000002}]},{"osd":4,"last update":"Tue Mar 12 03:21:54 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:22:12.331 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T03:22:12.331 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T03:22:12.331 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T03:22:12.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T03:22:12.537 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:12.920 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:12 smithi118 ceph-mon[29160]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:22:12.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:12 smithi177 ceph-mon[37140]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:22:13.905 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:13 smithi118 ceph-mon[29160]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:13.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:13 smithi177 ceph-mon[37140]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:14.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 -- 172.21.15.118:0/637740891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 msgr2=0x7f6dcc0ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:14.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 --2- 172.21.15.118:0/637740891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc0ff670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009910 tx=0x7f6dbc009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:14.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 -- 172.21.15.118:0/637740891 shutdown_connections 2024-03-12T03:22:14.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 --2- 172.21.15.118:0/637740891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc0ff670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:14.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 --2- 172.21.15.118:0/637740891 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc106b10 0x7f6dcc106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:14.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 -- 172.21.15.118:0/637740891 >> 172.21.15.118:0/637740891 conn(0x7f6dcc074c50 msgr2=0x7f6dcc075060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:14.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 -- 172.21.15.118:0/637740891 shutdown_connections 2024-03-12T03:22:14.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.436+0000 7f6dd389d700 1 -- 172.21.15.118:0/637740891 wait complete. 2024-03-12T03:22:14.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd389d700 1 Processor -- start 2024-03-12T03:22:14.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd389d700 1 -- start start 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd389d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd389d700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc106b10 0x7f6dcc19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd389d700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc19d290 con 0x7f6dcc0ff210 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd389d700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc196800 con 0x7f6dcc106b10 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd1639700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd1639700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44362/0 (socket says 172.21.15.118:44362) 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd1639700 1 -- 172.21.15.118:0/3956798871 learned_addr learned my addr 172.21.15.118:0/3956798871 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd0e38700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc106b10 0x7f6dcc19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd1639700 1 -- 172.21.15.118:0/3956798871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc106b10 msgr2=0x7f6dcc19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd1639700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc106b10 0x7f6dcc19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd1639700 1 -- 172.21.15.118:0/3956798871 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc0095f0 con 0x7f6dcc0ff210 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd0e38700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc106b10 0x7f6dcc19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.437+0000 7f6dd1639700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc19c6c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800d8d0 tx=0x7f6dc800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:14.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.438+0000 7f6dc27fc700 1 -- 172.21.15.118:0/3956798871 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc8009940 con 0x7f6dcc0ff210 2024-03-12T03:22:14.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.438+0000 7f6dc27fc700 1 -- 172.21.15.118:0/3956798871 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dc8010460 con 0x7f6dcc0ff210 2024-03-12T03:22:14.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.438+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc196ae0 con 0x7f6dcc0ff210 2024-03-12T03:22:14.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.438+0000 7f6dc27fc700 1 -- 172.21.15.118:0/3956798871 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6dc800f5f0 con 0x7f6dcc0ff210 2024-03-12T03:22:14.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.438+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc197030 con 0x7f6dcc0ff210 2024-03-12T03:22:14.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.439+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc109010 con 0x7f6dcc0ff210 2024-03-12T03:22:14.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.439+0000 7f6dc27fc700 1 -- 172.21.15.118:0/3956798871 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6dc80105d0 con 0x7f6dcc0ff210 2024-03-12T03:22:14.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.439+0000 7f6dc27fc700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6db806c880 0x7f6db806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:14.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.440+0000 7f6dc27fc700 1 -- 172.21.15.118:0/3956798871 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f6dc808b950 con 0x7f6dcc0ff210 2024-03-12T03:22:14.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.440+0000 7f6dd0e38700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6db806c880 0x7f6db806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:14.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.440+0000 7f6dd0e38700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6db806c880 0x7f6db806ed40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc0059d0 tx=0x7f6dbc01a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:14.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.444+0000 7f6dc27fc700 1 -- 172.21.15.118:0/3956798871 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6dc8059d30 con 0x7f6dcc0ff210 2024-03-12T03:22:14.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.680+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6dcc068c00 con 0x7f6dcc0ff210 2024-03-12T03:22:14.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.680+0000 7f6dc27fc700 1 -- 172.21.15.118:0/3956798871 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6dc80598c0 con 0x7f6dcc0ff210 2024-03-12T03:22:14.683 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:14.683 INFO:teuthology.orchestra.run.smithi118.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T03:22:14.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6db806c880 msgr2=0x7f6db806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:14.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6db806c880 0x7f6db806ed40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc0059d0 tx=0x7f6dbc01a040 comp rx=0 tx=0).stop 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 msgr2=0x7f6dcc19c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc19c6c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800d8d0 tx=0x7f6dc800dc90 comp rx=0 tx=0).stop 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 shutdown_connections 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f6db806c880 0x7f6db806ed40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6dcc0ff210 0x7f6dcc19c6c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 --2- 172.21.15.118:0/3956798871 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f6dcc106b10 0x7f6dcc19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 >> 172.21.15.118:0/3956798871 conn(0x7f6dcc074c50 msgr2=0x7f6dcc105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 shutdown_connections 2024-03-12T03:22:14.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:14.682+0000 7f6dd389d700 1 -- 172.21.15.118:0/3956798871 wait complete. 2024-03-12T03:22:14.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:14 smithi118 ceph-mon[29160]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:14.966 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:14 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3956798871' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:22:15.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:14 smithi177 ceph-mon[37140]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:15.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:14 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3956798871' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:22:15.312 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T03:22:15.312 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T03:22:15.313 INFO:teuthology.run_tasks:Running task print... 2024-03-12T03:22:15.323 INFO:teuthology.task.print:**** done end installing reef cephadm ... 2024-03-12T03:22:15.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:22:15.332 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:22:15.333 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-12T03:22:15.553 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:17.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2010402428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb38100b80 msgr2=0x7ffb38100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:17.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2010402428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb38100b80 0x7ffb38100fe0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ffb24009910 tx=0x7ffb24009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:17.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2010402428 shutdown_connections 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2010402428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb38100b80 0x7ffb38100fe0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2010402428 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb380ff980 0x7ffb380ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2010402428 >> 172.21.15.118:0/2010402428 conn(0x7ffb380faf60 msgr2=0x7ffb380fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2010402428 shutdown_connections 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.694+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2010402428 wait complete. 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3eeeb700 1 Processor -- start 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3eeeb700 1 -- start start 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3eeeb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb380ff980 0x7ffb381964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3eeeb700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb38100b80 0x7ffb381969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3eeeb700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38197010 con 0x7ffb380ff980 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3eeeb700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb38197150 con 0x7ffb38100b80 2024-03-12T03:22:17.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3cc87700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb380ff980 0x7ffb381964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3cc87700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb380ff980 0x7ffb381964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44384/0 (socket says 172.21.15.118:44384) 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3cc87700 1 -- 172.21.15.118:0/2327445195 learned_addr learned my addr 172.21.15.118:0/2327445195 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb37fff700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb38100b80 0x7ffb381969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3cc87700 1 -- 172.21.15.118:0/2327445195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb38100b80 msgr2=0x7ffb381969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3cc87700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb38100b80 0x7ffb381969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3cc87700 1 -- 172.21.15.118:0/2327445195 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb240095f0 con 0x7ffb380ff980 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3cc87700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb380ff980 0x7ffb381964b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00d900 tx=0x7ffb2c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb35ffb700 1 -- 172.21.15.118:0/2327445195 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb2c0041f0 con 0x7ffb380ff980 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb35ffb700 1 -- 172.21.15.118:0/2327445195 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb2c004350 con 0x7ffb380ff980 2024-03-12T03:22:17.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3819bb40 con 0x7ffb380ff980 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.695+0000 7ffb35ffb700 1 -- 172.21.15.118:0/2327445195 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb2c003cf0 con 0x7ffb380ff980 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.696+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3819c090 con 0x7ffb380ff980 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.697+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb380623e0 con 0x7ffb380ff980 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.697+0000 7ffb35ffb700 1 -- 172.21.15.118:0/2327445195 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ffb2c009730 con 0x7ffb380ff980 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.697+0000 7ffb35ffb700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ffb2806c880 0x7ffb2806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.697+0000 7ffb35ffb700 1 -- 172.21.15.118:0/2327445195 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7ffb2c08b7b0 con 0x7ffb380ff980 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.697+0000 7ffb37fff700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ffb2806c880 0x7ffb2806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.698+0000 7ffb37fff700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ffb2806c880 0x7ffb2806ed40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb24005090 tx=0x7ffb24005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:17.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.702+0000 7ffb35ffb700 1 -- 172.21.15.118:0/2327445195 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ffb2c059b90 con 0x7ffb380ff980 2024-03-12T03:22:17.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.894+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7ffb3819c730 con 0x7ffb380ff980 2024-03-12T03:22:17.897 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:17 smithi118 ceph-mon[29160]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:17.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.898+0000 7ffb35ffb700 1 -- 172.21.15.118:0/2327445195 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7ffb2c059720 con 0x7ffb380ff980 2024-03-12T03:22:17.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.900+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ffb2806c880 msgr2=0x7ffb2806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:17.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.900+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ffb2806c880 0x7ffb2806ed40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb24005090 tx=0x7ffb24005960 comp rx=0 tx=0).stop 2024-03-12T03:22:17.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.900+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb380ff980 msgr2=0x7ffb381964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:17.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.900+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb380ff980 0x7ffb381964b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00d900 tx=0x7ffb2c00dc10 comp rx=0 tx=0).stop 2024-03-12T03:22:17.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.901+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 shutdown_connections 2024-03-12T03:22:17.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.901+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ffb2806c880 0x7ffb2806ed40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:17.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.901+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb380ff980 0x7ffb381964b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:17.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.901+0000 7ffb3eeeb700 1 --2- 172.21.15.118:0/2327445195 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ffb38100b80 0x7ffb381969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:17.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.901+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 >> 172.21.15.118:0/2327445195 conn(0x7ffb380faf60 msgr2=0x7ffb38068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:17.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.901+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 shutdown_connections 2024-03-12T03:22:17.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:17.901+0000 7ffb3eeeb700 1 -- 172.21.15.118:0/2327445195 wait complete. 2024-03-12T03:22:17.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:17 smithi177 ceph-mon[37140]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:18.579 INFO:teuthology.run_tasks:Running task print... 2024-03-12T03:22:18.589 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-12T03:22:18.589 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:22:18.597 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:22:18.597 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-12T03:22:18.807 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:19.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:18 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2327445195' entity='client.admin' 2024-03-12T03:22:19.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:19.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:18 smithi118 ceph-mon[29160]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:19.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:19.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:19.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:19.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:19.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:18 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2327445195' entity='client.admin' 2024-03-12T03:22:19.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:19.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:18 smithi177 ceph-mon[37140]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:19.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:19.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:19.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 -- 172.21.15.118:0/368524811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 msgr2=0x7f7da40ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 --2- 172.21.15.118:0/368524811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da40ffda0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7d98009910 tx=0x7f7d98009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 -- 172.21.15.118:0/368524811 shutdown_connections 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 --2- 172.21.15.118:0/368524811 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7da4100b80 0x7f7da4100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 --2- 172.21.15.118:0/368524811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da40ffda0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 -- 172.21.15.118:0/368524811 >> 172.21.15.118:0/368524811 conn(0x7f7da40faf60 msgr2=0x7f7da40fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 -- 172.21.15.118:0/368524811 shutdown_connections 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.308+0000 7f7daae48700 1 -- 172.21.15.118:0/368524811 wait complete. 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7daae48700 1 Processor -- start 2024-03-12T03:22:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7daae48700 1 -- start start 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7daae48700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da4196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7daae48700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7da4100b80 0x7f7da41969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7daae48700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da4196ff0 con 0x7f7da40ff980 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7daae48700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da4197130 con 0x7f7da4100b80 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7da8be4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da4196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7da8be4700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da4196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44404/0 (socket says 172.21.15.118:44404) 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7da8be4700 1 -- 172.21.15.118:0/88708442 learned_addr learned my addr 172.21.15.118:0/88708442 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7da8be4700 1 -- 172.21.15.118:0/88708442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7da4100b80 msgr2=0x7f7da41969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7da8be4700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7da4100b80 0x7f7da41969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7da8be4700 1 -- 172.21.15.118:0/88708442 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d980095f0 con 0x7f7da40ff980 2024-03-12T03:22:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.309+0000 7f7da8be4700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da4196490 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7d980059d0 tx=0x7f7d9800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:20.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.310+0000 7f7da1ffb700 1 -- 172.21.15.118:0/88708442 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d9801d070 con 0x7f7da40ff980 2024-03-12T03:22:20.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.310+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da419bb30 con 0x7f7da40ff980 2024-03-12T03:22:20.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.310+0000 7f7da1ffb700 1 -- 172.21.15.118:0/88708442 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7d9800f460 con 0x7f7da40ff980 2024-03-12T03:22:20.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.310+0000 7f7da1ffb700 1 -- 172.21.15.118:0/88708442 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d98021640 con 0x7f7da40ff980 2024-03-12T03:22:20.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.310+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da419c0a0 con 0x7f7da40ff980 2024-03-12T03:22:20.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.311+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da40623e0 con 0x7f7da40ff980 2024-03-12T03:22:20.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.311+0000 7f7da1ffb700 1 -- 172.21.15.118:0/88708442 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7d9800f5d0 con 0x7f7da40ff980 2024-03-12T03:22:20.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.312+0000 7f7da1ffb700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d9406c880 0x7f7d9406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:20.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.312+0000 7f7da1ffb700 1 -- 172.21.15.118:0/88708442 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f7d9808cc10 con 0x7f7da40ff980 2024-03-12T03:22:20.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.312+0000 7f7da3fff700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d9406c880 0x7f7d9406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:20.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.312+0000 7f7da3fff700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d9406c880 0x7f7d9406ed40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000a990 tx=0x7f7d90005c80 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:20.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.316+0000 7f7da1ffb700 1 -- 172.21.15.118:0/88708442 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7d9805b070 con 0x7f7da40ff980 2024-03-12T03:22:20.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.525+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7da4068b10 con 0x7f7d9406c880 2024-03-12T03:22:20.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.526+0000 7f7da1ffb700 1 -- 172.21.15.118:0/88708442 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7da4068b10 con 0x7f7d9406c880 2024-03-12T03:22:20.528 INFO:teuthology.orchestra.run.smithi118.stdout:Backend: cephadm 2024-03-12T03:22:20.528 INFO:teuthology.orchestra.run.smithi118.stdout:Available: Yes 2024-03-12T03:22:20.528 INFO:teuthology.orchestra.run.smithi118.stdout:Paused: No 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d9406c880 msgr2=0x7f7d9406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d9406c880 0x7f7d9406ed40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000a990 tx=0x7f7d90005c80 comp rx=0 tx=0).stop 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 msgr2=0x7f7da4196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da4196490 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7d980059d0 tx=0x7f7d9800be20 comp rx=0 tx=0).stop 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 shutdown_connections 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7d9406c880 0x7f7d9406ed40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7da40ff980 0x7f7da4196490 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 --2- 172.21.15.118:0/88708442 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7da4100b80 0x7f7da41969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:20.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 >> 172.21.15.118:0/88708442 conn(0x7f7da40faf60 msgr2=0x7f7da4068400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:20.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 shutdown_connections 2024-03-12T03:22:20.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:20.528+0000 7f7daae48700 1 -- 172.21.15.118:0/88708442 wait complete. 2024-03-12T03:22:21.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-12T03:22:21.491 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:21.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:21 smithi177 ceph-mon[37140]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:21.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:21 smithi177 ceph-mon[37140]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:22.001 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:21 smithi118 ceph-mon[29160]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:22.002 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:21 smithi118 ceph-mon[29160]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 -- 172.21.15.118:0/1078845632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 msgr2=0x7fc8ac100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 --2- 172.21.15.118:0/1078845632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac100fe0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009910 tx=0x7fc8a0009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 -- 172.21.15.118:0/1078845632 shutdown_connections 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 --2- 172.21.15.118:0/1078845632 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac100fe0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 --2- 172.21.15.118:0/1078845632 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8ac0ff980 0x7fc8ac0ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 -- 172.21.15.118:0/1078845632 >> 172.21.15.118:0/1078845632 conn(0x7fc8ac0faf60 msgr2=0x7fc8ac0fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 -- 172.21.15.118:0/1078845632 shutdown_connections 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.148+0000 7fc8b2193700 1 -- 172.21.15.118:0/1078845632 wait complete. 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8b2193700 1 Processor -- start 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8b2193700 1 -- start start 2024-03-12T03:22:23.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8b2193700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8ac0ff980 0x7fc8ac195360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8b2193700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac1958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8b2193700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac195ec0 con 0x7fc8ac100b80 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8b2193700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac196000 con 0x7fc8ac0ff980 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8aaffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac1958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8aaffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac1958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44422/0 (socket says 172.21.15.118:44422) 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8aaffd700 1 -- 172.21.15.118:0/2312468362 learned_addr learned my addr 172.21.15.118:0/2312468362 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8ab7fe700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8ac0ff980 0x7fc8ac195360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8aaffd700 1 -- 172.21.15.118:0/2312468362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8ac0ff980 msgr2=0x7fc8ac195360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8aaffd700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8ac0ff980 0x7fc8ac195360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8aaffd700 1 -- 172.21.15.118:0/2312468362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a00095f0 con 0x7fc8ac100b80 2024-03-12T03:22:23.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.149+0000 7fc8ab7fe700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8ac0ff980 0x7fc8ac195360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T03:22:23.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.150+0000 7fc8aaffd700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac1958a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00092e0 tx=0x7fc8a000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:23.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.150+0000 7fc8a8ff9700 1 -- 172.21.15.118:0/2312468362 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a001d070 con 0x7fc8ac100b80 2024-03-12T03:22:23.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.150+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ac19aa00 con 0x7fc8ac100b80 2024-03-12T03:22:23.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.150+0000 7fc8a8ff9700 1 -- 172.21.15.118:0/2312468362 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8a000f460 con 0x7fc8ac100b80 2024-03-12T03:22:23.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.150+0000 7fc8a8ff9700 1 -- 172.21.15.118:0/2312468362 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a0021640 con 0x7fc8ac100b80 2024-03-12T03:22:23.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.150+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ac19af70 con 0x7fc8ac100b80 2024-03-12T03:22:23.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.151+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac18f500 con 0x7fc8ac100b80 2024-03-12T03:22:23.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.151+0000 7fc8a8ff9700 1 -- 172.21.15.118:0/2312468362 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc8a000fad0 con 0x7fc8ac100b80 2024-03-12T03:22:23.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.152+0000 7fc8a8ff9700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc89406c880 0x7fc89406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:23.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.152+0000 7fc8ab7fe700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc89406c880 0x7fc89406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:23.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.152+0000 7fc8a8ff9700 1 -- 172.21.15.118:0/2312468362 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fc8a008cb60 con 0x7fc8ac100b80 2024-03-12T03:22:23.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.152+0000 7fc8ab7fe700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc89406c880 0x7fc89406ed40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc89c005fd0 tx=0x7fc89c005f00 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:23.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.156+0000 7fc8a8ff9700 1 -- 172.21.15.118:0/2312468362 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc8a005af40 con 0x7fc8ac100b80 2024-03-12T03:22:23.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.358+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8ac0611f0 con 0x7fc89406c880 2024-03-12T03:22:23.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.367+0000 7fc8a8ff9700 1 -- 172.21.15.118:0/2312468362 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3080 (secure 0 0 0) 0x7fc8ac0611f0 con 0x7fc89406c880 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:alertmanager.smithi118 smithi118 *:9093,9094 running (3m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 ab6c8c340db8 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-exporter.smithi118 smithi118 running (5m) 2m ago 5m 7054k - 18.2.2-751-g62a11b3c fa4cc2e02803 dab7493ee414 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-exporter.smithi177 smithi177 running (4m) 65s ago 4m 6979k - 18.2.2-751-g62a11b3c fa4cc2e02803 56905fbf68ab 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:crash.smithi118 smithi118 running (5m) 2m ago 5m 6660k - 18.2.2-751-g62a11b3c fa4cc2e02803 848a90530bcc 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:crash.smithi177 smithi177 running (4m) 65s ago 4m 6660k - 18.2.2-751-g62a11b3c fa4cc2e02803 73a879109cfd 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:grafana.smithi118 smithi118 *:3000 running (3m) 2m ago 4m 77.3M - 9.4.7 954c08fa6188 f56cb9df4974 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.smithi118.zzzuvl smithi118 *:9283,8765,8443 running (6m) 2m ago 6m 489M - 18.2.2-751-g62a11b3c fa4cc2e02803 b31fb006922b 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.smithi177.tmmsuz smithi177 *:8443,9283,8765 running (4m) 65s ago 4m 452M - 18.2.2-751-g62a11b3c fa4cc2e02803 9f6d8c95b283 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:mon.smithi118 smithi118 running (6m) 2m ago 6m 43.7M 2048M 18.2.2-751-g62a11b3c fa4cc2e02803 a26ddd75d886 2024-03-12T03:22:23.370 INFO:teuthology.orchestra.run.smithi118.stdout:mon.smithi177 smithi177 running (4m) 65s ago 4m 41.7M 2048M 18.2.2-751-g62a11b3c fa4cc2e02803 0d8acd4f44b3 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:node-exporter.smithi118 smithi118 *:9100 running (5m) 2m ago 5m 16.3M - 1.5.0 0da6a335fe13 4e2acb3f4a85 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:node-exporter.smithi177 smithi177 *:9100 running (4m) 65s ago 4m 16.9M - 1.5.0 0da6a335fe13 bde9c5ddfe9c 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:osd.0 smithi118 running (3m) 2m ago 3m 43.5M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 9436f155422d 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:osd.1 smithi118 running (2m) 2m ago 2m 41.0M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 38fd187fe41b 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:osd.2 smithi118 running (2m) 2m ago 2m 39.4M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 a5856da80958 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:osd.3 smithi177 running (2m) 65s ago 2m 41.9M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 1a594e7be73e 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:osd.4 smithi177 running (97s) 65s ago 96s 41.8M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 a4b1d7721078 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:osd.5 smithi177 running (70s) 65s ago 70s 10.5M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 fd4d4398ebeb 2024-03-12T03:22:23.371 INFO:teuthology.orchestra.run.smithi118.stdout:prometheus.smithi118 smithi118 *:9095 running (3m) 2m ago 4m 37.5M - 2.43.0 a07b618ecd1d 367c172183f1 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc89406c880 msgr2=0x7fc89406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc89406c880 0x7fc89406ed40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc89c005fd0 tx=0x7fc89c005f00 comp rx=0 tx=0).stop 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 msgr2=0x7fc8ac1958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac1958a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00092e0 tx=0x7fc8a000be20 comp rx=0 tx=0).stop 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 shutdown_connections 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc89406c880 0x7fc89406ed40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8ac0ff980 0x7fc8ac195360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 --2- 172.21.15.118:0/2312468362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8ac100b80 0x7fc8ac1958a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:23.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.369+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 >> 172.21.15.118:0/2312468362 conn(0x7fc8ac0faf60 msgr2=0x7fc8ac104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:23.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.370+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 shutdown_connections 2024-03-12T03:22:23.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:23.370+0000 7fc8b2193700 1 -- 172.21.15.118:0/2312468362 wait complete. 2024-03-12T03:22:23.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:23 smithi177 ceph-mon[37140]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:24.005 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:23 smithi118 ceph-mon[29160]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:24.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-12T03:22:24.232 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:24.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:24 smithi177 ceph-mon[37140]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:25.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:24 smithi118 ceph-mon[29160]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:25.908 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:25 smithi118 ceph-mon[29160]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 -- 172.21.15.118:0/896856364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 msgr2=0x7fb3b81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 --2- 172.21.15.118:0/896856364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b81039a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009910 tx=0x7fb3a8009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 -- 172.21.15.118:0/896856364 shutdown_connections 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 --2- 172.21.15.118:0/896856364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3b8103ee0 0x7fb3b81062d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 --2- 172.21.15.118:0/896856364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b81039a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 -- 172.21.15.118:0/896856364 >> 172.21.15.118:0/896856364 conn(0x7fb3b80faf60 msgr2=0x7fb3b80fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 -- 172.21.15.118:0/896856364 shutdown_connections 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 -- 172.21.15.118:0/896856364 wait complete. 2024-03-12T03:22:25.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.908+0000 7fb3bd359700 1 Processor -- start 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3bd359700 1 -- start start 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3bd359700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b8196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3bd359700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3b8103ee0 0x7fb3b8196aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3bd359700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81970c0 con 0x7fb3b81015b0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3bd359700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b8197200 con 0x7fb3b8103ee0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b8196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b8196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:59616/0 (socket says 172.21.15.118:59616) 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b6ffd700 1 -- 172.21.15.118:0/701868058 learned_addr learned my addr 172.21.15.118:0/701868058 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b67fc700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3b8103ee0 0x7fb3b8196aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b6ffd700 1 -- 172.21.15.118:0/701868058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3b8103ee0 msgr2=0x7fb3b8196aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b6ffd700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3b8103ee0 0x7fb3b8196aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b6ffd700 1 -- 172.21.15.118:0/701868058 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a80095f0 con 0x7fb3b81015b0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb3b6ffd700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b8196560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80092e0 tx=0x7fb3a800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.909+0000 7fb39ffff700 1 -- 172.21.15.118:0/701868058 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a801d070 con 0x7fb3b81015b0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.910+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b819bc00 con 0x7fb3b81015b0 2024-03-12T03:22:25.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.910+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b819c0f0 con 0x7fb3b81015b0 2024-03-12T03:22:25.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.910+0000 7fb39ffff700 1 -- 172.21.15.118:0/701868058 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3a800f460 con 0x7fb3b81015b0 2024-03-12T03:22:25.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.910+0000 7fb39ffff700 1 -- 172.21.15.118:0/701868058 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3a8021640 con 0x7fb3b81015b0 2024-03-12T03:22:25.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.911+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b80623e0 con 0x7fb3b81015b0 2024-03-12T03:22:25.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.911+0000 7fb39ffff700 1 -- 172.21.15.118:0/701868058 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fb3a802b440 con 0x7fb3b81015b0 2024-03-12T03:22:25.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.911+0000 7fb39ffff700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fb3a006c880 0x7fb3a006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:25.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.912+0000 7fb3b67fc700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fb3a006c880 0x7fb3a006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:25.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.912+0000 7fb39ffff700 1 -- 172.21.15.118:0/701868058 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7fb3a808cd10 con 0x7fb3b81015b0 2024-03-12T03:22:25.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.912+0000 7fb3b67fc700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fb3a006c880 0x7fb3a006ed40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac006fd0 tx=0x7fb3ac008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:25.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:25.916+0000 7fb39ffff700 1 -- 172.21.15.118:0/701868058 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fb3a805b0f0 con 0x7fb3b81015b0 2024-03-12T03:22:25.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:25 smithi177 ceph-mon[37140]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:26.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.117+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3b8104920 con 0x7fb3a006c880 2024-03-12T03:22:26.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.121+0000 7fb39ffff700 1 -- 172.21.15.118:0/701868058 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1310 (secure 0 0 0) 0x7fb3b8104920 con 0x7fb3a006c880 2024-03-12T03:22:26.123 INFO:teuthology.orchestra.run.smithi118.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-exporter 2/2 2m ago 6m * 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:crash 2/2 2m ago 6m * 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:mgr 2/2 2m ago 6m count:2 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:mon 2/2 2m ago 5m smithi118:172.21.15.118=smithi118;smithi177:172.21.15.177=smithi177;count:2 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:osd 6 2m ago - 2024-03-12T03:22:26.124 INFO:teuthology.orchestra.run.smithi118.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fb3a006c880 msgr2=0x7fb3a006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fb3a006c880 0x7fb3a006ed40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac006fd0 tx=0x7fb3ac008040 comp rx=0 tx=0).stop 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 msgr2=0x7fb3b8196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b8196560 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80092e0 tx=0x7fb3a800be20 comp rx=0 tx=0).stop 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 shutdown_connections 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fb3a006c880 0x7fb3a006ed40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3b81015b0 0x7fb3b8196560 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 --2- 172.21.15.118:0/701868058 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb3b8103ee0 0x7fb3b8196aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 >> 172.21.15.118:0/701868058 conn(0x7fb3b80faf60 msgr2=0x7fb3b80fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 shutdown_connections 2024-03-12T03:22:26.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:26.123+0000 7fb3bd359700 1 -- 172.21.15.118:0/701868058 wait complete. 2024-03-12T03:22:26.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-12T03:22:26.941 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:27.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:27 smithi177 ceph-mon[37140]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:27.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:27 smithi177 ceph-mon[37140]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:28.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:27 smithi118 ceph-mon[29160]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:28.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:27 smithi118 ceph-mon[29160]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 -- 172.21.15.118:0/570802474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 msgr2=0x7f28840ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 --2- 172.21.15.118:0/570802474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28840ffde0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2878009910 tx=0x7f2878009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 -- 172.21.15.118:0/570802474 shutdown_connections 2024-03-12T03:22:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 --2- 172.21.15.118:0/570802474 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2884100bc0 0x7f2884101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 --2- 172.21.15.118:0/570802474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28840ffde0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 -- 172.21.15.118:0/570802474 >> 172.21.15.118:0/570802474 conn(0x7f28840faf80 msgr2=0x7f28840fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:28.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 -- 172.21.15.118:0/570802474 shutdown_connections 2024-03-12T03:22:28.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.613+0000 7f288af85700 1 -- 172.21.15.118:0/570802474 wait complete. 2024-03-12T03:22:28.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f288af85700 1 Processor -- start 2024-03-12T03:22:28.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f288af85700 1 -- start start 2024-03-12T03:22:28.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f288af85700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28841965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:28.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f288af85700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2884100bc0 0x7f2884196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:28.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f288af85700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884197150 con 0x7f28840ff9c0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f288af85700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2884197290 con 0x7f2884100bc0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f2888d21700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28841965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f2888d21700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28841965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:59624/0 (socket says 172.21.15.118:59624) 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f2888d21700 1 -- 172.21.15.118:0/1551063500 learned_addr learned my addr 172.21.15.118:0/1551063500 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.614+0000 7f2883fff700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2884100bc0 0x7f2884196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f2888d21700 1 -- 172.21.15.118:0/1551063500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2884100bc0 msgr2=0x7f2884196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f2888d21700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2884100bc0 0x7f2884196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f2888d21700 1 -- 172.21.15.118:0/1551063500 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780095f0 con 0x7f28840ff9c0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f2888d21700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28841965f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2878005850 tx=0x7f287800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f2881ffb700 1 -- 172.21.15.118:0/1551063500 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f287801d070 con 0x7f28840ff9c0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288419bc90 con 0x7f28840ff9c0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f2881ffb700 1 -- 172.21.15.118:0/1551063500 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f287800f460 con 0x7f28840ff9c0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f2881ffb700 1 -- 172.21.15.118:0/1551063500 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2878017610 con 0x7f28840ff9c0 2024-03-12T03:22:28.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.615+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288419c180 con 0x7f28840ff9c0 2024-03-12T03:22:28.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.616+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288404ea80 con 0x7f28840ff9c0 2024-03-12T03:22:28.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.617+0000 7f2881ffb700 1 -- 172.21.15.118:0/1551063500 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f287800f5d0 con 0x7f28840ff9c0 2024-03-12T03:22:28.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.617+0000 7f2881ffb700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f287406c880 0x7f287406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:28.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.617+0000 7f2881ffb700 1 -- 172.21.15.118:0/1551063500 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f287808cf00 con 0x7f28840ff9c0 2024-03-12T03:22:28.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.617+0000 7f2883fff700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f287406c880 0x7f287406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:28.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.617+0000 7f2883fff700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f287406c880 0x7f287406ed40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2870006fd0 tx=0x7f2870008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:28.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.622+0000 7f2881ffb700 1 -- 172.21.15.118:0/1551063500 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f287805b2e0 con 0x7f28840ff9c0 2024-03-12T03:22:28.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.820+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2884105510 con 0x7f287406c880 2024-03-12T03:22:28.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.821+0000 7f2881ffb700 1 -- 172.21.15.118:0/1551063500 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f2884105510 con 0x7f287406c880 2024-03-12T03:22:28.824 INFO:teuthology.orchestra.run.smithi118.stdout:HOST ADDR LABELS STATUS 2024-03-12T03:22:28.824 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118 172.21.15.118 2024-03-12T03:22:28.824 INFO:teuthology.orchestra.run.smithi118.stdout:smithi177 172.21.15.177 2024-03-12T03:22:28.824 INFO:teuthology.orchestra.run.smithi118.stdout:2 hosts in cluster 2024-03-12T03:22:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f287406c880 msgr2=0x7f287406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f287406c880 0x7f287406ed40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2870006fd0 tx=0x7f2870008040 comp rx=0 tx=0).stop 2024-03-12T03:22:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 msgr2=0x7f28841965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28841965f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2878005850 tx=0x7f287800be20 comp rx=0 tx=0).stop 2024-03-12T03:22:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 shutdown_connections 2024-03-12T03:22:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f287406c880 0x7f287406ed40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f28840ff9c0 0x7f28841965f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:28.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 --2- 172.21.15.118:0/1551063500 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2884100bc0 0x7f2884196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:28.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 >> 172.21.15.118:0/1551063500 conn(0x7f28840faf80 msgr2=0x7f2884103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:28.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 shutdown_connections 2024-03-12T03:22:28.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:28.823+0000 7f288af85700 1 -- 172.21.15.118:0/1551063500 wait complete. 2024-03-12T03:22:29.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:29 smithi118 ceph-mon[29160]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:29.875 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-12T03:22:29.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:29 smithi177 ceph-mon[37140]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:30.086 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:30.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:30 smithi118 ceph-mon[29160]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:30.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:30 smithi177 ceph-mon[37140]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:31.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:31 smithi177 ceph-mon[37140]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:31.979 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:31 smithi118 ceph-mon[29160]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 -- 172.21.15.118:0/4041361596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89500ff9c0 msgr2=0x7f89500ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 --2- 172.21.15.118:0/4041361596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89500ff9c0 0x7f89500ffde0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f894c009910 tx=0x7f894c009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 -- 172.21.15.118:0/4041361596 shutdown_connections 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 --2- 172.21.15.118:0/4041361596 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f8950100bc0 0x7f8950101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 --2- 172.21.15.118:0/4041361596 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89500ff9c0 0x7f89500ffde0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 -- 172.21.15.118:0/4041361596 >> 172.21.15.118:0/4041361596 conn(0x7f89500faf80 msgr2=0x7f89500fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 -- 172.21.15.118:0/4041361596 shutdown_connections 2024-03-12T03:22:32.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.197+0000 7f8957675700 1 -- 172.21.15.118:0/4041361596 wait complete. 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.198+0000 7f8957675700 1 Processor -- start 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.198+0000 7f8957675700 1 -- start start 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.198+0000 7f8957675700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89500ff9c0 0x7f8950195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.198+0000 7f8957675700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8950100bc0 0x7f8950195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.198+0000 7f8957675700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8950195f10 con 0x7f8950100bc0 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.198+0000 7f8957675700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8950196050 con 0x7f89500ff9c0 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.198+0000 7f8954c10700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8950100bc0 0x7f8950195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8954c10700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8950100bc0 0x7f8950195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:59644/0 (socket says 172.21.15.118:59644) 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8954c10700 1 -- 172.21.15.118:0/1160833378 learned_addr learned my addr 172.21.15.118:0/1160833378 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8955411700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89500ff9c0 0x7f8950195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:32.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8954c10700 1 -- 172.21.15.118:0/1160833378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89500ff9c0 msgr2=0x7f8950195440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8954c10700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89500ff9c0 0x7f8950195440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8954c10700 1 -- 172.21.15.118:0/1160833378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c0095f0 con 0x7f8950100bc0 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8954c10700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8950100bc0 0x7f8950195980 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f894000c930 tx=0x7f894000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f89467fc700 1 -- 172.21.15.118:0/1160833378 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8940007ad0 con 0x7f8950100bc0 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f89467fc700 1 -- 172.21.15.118:0/1160833378 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f894000ce80 con 0x7f8950100bc0 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895019aa70 con 0x7f8950100bc0 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f89467fc700 1 -- 172.21.15.118:0/1160833378 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8940018630 con 0x7f8950100bc0 2024-03-12T03:22:32.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.199+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895019b010 con 0x7f8950100bc0 2024-03-12T03:22:32.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.200+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895018f540 con 0x7f8950100bc0 2024-03-12T03:22:32.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.201+0000 7f89467fc700 1 -- 172.21.15.118:0/1160833378 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f8940007c30 con 0x7f8950100bc0 2024-03-12T03:22:32.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.201+0000 7f89467fc700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f893c06c880 0x7f893c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:32.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.201+0000 7f89467fc700 1 -- 172.21.15.118:0/1160833378 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f894008b930 con 0x7f8950100bc0 2024-03-12T03:22:32.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.201+0000 7f8955411700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f893c06c880 0x7f893c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:32.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.201+0000 7f8955411700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f893c06c880 0x7f893c06ed40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b5c0 tx=0x7f894c01a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:32.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.206+0000 7f89467fc700 1 -- 172.21.15.118:0/1160833378 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f8940059d10 con 0x7f8950100bc0 2024-03-12T03:22:32.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.407+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f89500611f0 con 0x7f893c06c880 2024-03-12T03:22:32.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.409+0000 7f89467fc700 1 -- 172.21.15.118:0/1160833378 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f89500611f0 con 0x7f893c06c880 2024-03-12T03:22:32.411 INFO:teuthology.orchestra.run.smithi118.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-12T03:22:32.412 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747500JS480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-12T03:22:32.412 INFO:teuthology.orchestra.run.smithi118.stdout:smithi177 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233007S400BGN 372G No 70s ago Has a FileSystem, LVM detected 2024-03-12T03:22:32.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f893c06c880 msgr2=0x7f893c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:32.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f893c06c880 0x7f893c06ed40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f894c00b5c0 tx=0x7f894c01a040 comp rx=0 tx=0).stop 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8950100bc0 msgr2=0x7f8950195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8950100bc0 0x7f8950195980 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f894000c930 tx=0x7f894000cc40 comp rx=0 tx=0).stop 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 shutdown_connections 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f893c06c880 0x7f893c06ed40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f89500ff9c0 0x7f8950195440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 --2- 172.21.15.118:0/1160833378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8950100bc0 0x7f8950195980 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 >> 172.21.15.118:0/1160833378 conn(0x7f89500faf80 msgr2=0x7f8950104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 shutdown_connections 2024-03-12T03:22:32.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:32.411+0000 7f8957675700 1 -- 172.21.15.118:0/1160833378 wait complete. 2024-03-12T03:22:33.549 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:22:33.566 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:22:33.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-12T03:22:33.775 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:33.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:33 smithi118 ceph-mon[29160]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:33.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:33 smithi118 ceph-mon[29160]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:33.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:33 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:33.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:33 smithi177 ceph-mon[37140]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:33.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:33 smithi177 ceph-mon[37140]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:33 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:35.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:34 smithi177 ceph-mon[37140]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:35.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:34 smithi118 ceph-mon[29160]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:35.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 -- 172.21.15.118:0/109283373 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 msgr2=0x7f1d98106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:35.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 --2- 172.21.15.118:0/109283373 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98106230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d80009910 tx=0x7f1d80009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:35.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 -- 172.21.15.118:0/109283373 shutdown_connections 2024-03-12T03:22:35.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 --2- 172.21.15.118:0/109283373 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98106230 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:35.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 --2- 172.21.15.118:0/109283373 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d98101510 0x7f1d98103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 -- 172.21.15.118:0/109283373 >> 172.21.15.118:0/109283373 conn(0x7f1d980faf60 msgr2=0x7f1d980fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 -- 172.21.15.118:0/109283373 shutdown_connections 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.883+0000 7f1d9c944700 1 -- 172.21.15.118:0/109283373 wait complete. 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d9c944700 1 Processor -- start 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d9c944700 1 -- start start 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d9c944700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d98101510 0x7f1d98196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d9c944700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98196b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d9c944700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d98197170 con 0x7f1d98103e40 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d9c944700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d981972b0 con 0x7f1d98101510 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d95d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98196b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d95d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98196b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60098/0 (socket says 172.21.15.118:60098) 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d95d9b700 1 -- 172.21.15.118:0/1755150112 learned_addr learned my addr 172.21.15.118:0/1755150112 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d9659c700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d98101510 0x7f1d98196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d95d9b700 1 -- 172.21.15.118:0/1755150112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d98101510 msgr2=0x7f1d98196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:35.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d95d9b700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d98101510 0x7f1d98196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:35.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.884+0000 7f1d95d9b700 1 -- 172.21.15.118:0/1755150112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d800095f0 con 0x7f1d98103e40 2024-03-12T03:22:35.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.885+0000 7f1d95d9b700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98196b50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d80006010 tx=0x7f1d8000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:35.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.885+0000 7f1d8f7fe700 1 -- 172.21.15.118:0/1755150112 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d8001d070 con 0x7f1d98103e40 2024-03-12T03:22:35.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.885+0000 7f1d8f7fe700 1 -- 172.21.15.118:0/1755150112 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d8000f460 con 0x7f1d98103e40 2024-03-12T03:22:35.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.885+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9819bc60 con 0x7f1d98103e40 2024-03-12T03:22:35.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.885+0000 7f1d8f7fe700 1 -- 172.21.15.118:0/1755150112 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d80021640 con 0x7f1d98103e40 2024-03-12T03:22:35.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.885+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9819c1d0 con 0x7f1d98103e40 2024-03-12T03:22:35.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.886+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d980623e0 con 0x7f1d98103e40 2024-03-12T03:22:35.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.886+0000 7f1d8f7fe700 1 -- 172.21.15.118:0/1755150112 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1d8000fad0 con 0x7f1d98103e40 2024-03-12T03:22:35.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.887+0000 7f1d8f7fe700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1d8406c880 0x7f1d8406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:35.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.887+0000 7f1d8f7fe700 1 -- 172.21.15.118:0/1755150112 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f1d8008cc90 con 0x7f1d98103e40 2024-03-12T03:22:35.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.887+0000 7f1d9659c700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1d8406c880 0x7f1d8406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:35.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.887+0000 7f1d9659c700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1d8406c880 0x7f1d8406ed40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009f20 tx=0x7f1d880094f0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:35.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:35.891+0000 7f1d8f7fe700 1 -- 172.21.15.118:0/1755150112 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1d8005b070 con 0x7f1d98103e40 2024-03-12T03:22:36.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:36.127+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d98104900 con 0x7f1d8406c880 2024-03-12T03:22:36.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:36 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-12T03:22:37.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:36 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-12T03:22:37.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.450+0000 7f1d8f7fe700 1 -- 172.21.15.118:0/1755150112 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1d98104900 con 0x7f1d8406c880 2024-03-12T03:22:37.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1d8406c880 msgr2=0x7f1d8406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:37.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1d8406c880 0x7f1d8406ed40 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1d88009f20 tx=0x7f1d880094f0 comp rx=0 tx=0).stop 2024-03-12T03:22:37.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 msgr2=0x7f1d98196b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:37.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98196b50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d80006010 tx=0x7f1d8000be20 comp rx=0 tx=0).stop 2024-03-12T03:22:37.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 shutdown_connections 2024-03-12T03:22:37.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1d8406c880 0x7f1d8406ed40 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:37.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1d98101510 0x7f1d98196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:37.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.452+0000 7f1d9c944700 1 --2- 172.21.15.118:0/1755150112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d98103e40 0x7f1d98196b50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:37.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.453+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 >> 172.21.15.118:0/1755150112 conn(0x7f1d980faf60 msgr2=0x7f1d980fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:37.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.453+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 shutdown_connections 2024-03-12T03:22:37.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:37.453+0000 7f1d9c944700 1 -- 172.21.15.118:0/1755150112 wait complete. 2024-03-12T03:22:37.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:22:37.425+0000 7f8eebaca700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T03:22:37.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:37.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: pgmap v174: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: osdmap e43: 6 total, 6 up, 6 in 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: osdmap e44: 6 total, 6 up, 6 in 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: fsmap cephfs:0 2024-03-12T03:22:37.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:37.813 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:37 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:22:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: pgmap v174: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: osdmap e43: 6 total, 6 up, 6 in 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: osdmap e44: 6 total, 6 up, 6 in 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: fsmap cephfs:0 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:37.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:37 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:38.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-12T03:22:38.780 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: Saving service mds.cephfs spec with placement count:4 2024-03-12T03:22:38.781 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:38.781 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:38.781 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:38.781 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.wouiai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:38.781 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.wouiai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:38.781 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:38.781 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:38 smithi118 ceph-mon[29160]: osdmap e45: 6 total, 6 up, 6 in 2024-03-12T03:22:38.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: Saving service mds.cephfs spec with placement count:4 2024-03-12T03:22:38.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:38.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:22:38.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:38.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.wouiai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:38.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.wouiai", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:38.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:38.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:38 smithi177 ceph-mon[37140]: osdmap e45: 6 total, 6 up, 6 in 2024-03-12T03:22:39.003 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:39.755 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:39 smithi118 ceph-mon[29160]: Deploying daemon mds.cephfs.smithi118.wouiai on smithi118 2024-03-12T03:22:39.755 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:39 smithi118 ceph-mon[29160]: pgmap v178: 65 pgs: 16 active+clean, 9 creating+peering, 40 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:39.755 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:39 smithi118 ceph-mon[29160]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:22:39.755 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:39 smithi118 ceph-mon[29160]: osdmap e46: 6 total, 6 up, 6 in 2024-03-12T03:22:39.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:39 smithi177 ceph-mon[37140]: Deploying daemon mds.cephfs.smithi118.wouiai on smithi118 2024-03-12T03:22:39.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:39 smithi177 ceph-mon[37140]: pgmap v178: 65 pgs: 16 active+clean, 9 creating+peering, 40 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:39.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:39 smithi177 ceph-mon[37140]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-12T03:22:39.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:39 smithi177 ceph-mon[37140]: osdmap e46: 6 total, 6 up, 6 in 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 -- 172.21.15.118:0/2579780461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 msgr2=0x7f1a2c102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 --2- 172.21.15.118:0/2579780461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c102110 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1a20009910 tx=0x7f1a20009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 -- 172.21.15.118:0/2579780461 shutdown_connections 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 --2- 172.21.15.118:0/2579780461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c102110 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 --2- 172.21.15.118:0/2579780461 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a2c100ab0 0x7f1a2c100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 -- 172.21.15.118:0/2579780461 >> 172.21.15.118:0/2579780461 conn(0x7f1a2c0fc090 msgr2=0x7f1a2c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 -- 172.21.15.118:0/2579780461 shutdown_connections 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.072+0000 7f1a32365700 1 -- 172.21.15.118:0/2579780461 wait complete. 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a32365700 1 Processor -- start 2024-03-12T03:22:41.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a32365700 1 -- start start 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a32365700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a2c100ab0 0x7f1a2c196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a32365700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a32365700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c197160 con 0x7f1a2c101cb0 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a32365700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c1972a0 con 0x7f1a2c100ab0 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a2b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a2b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c196b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60126/0 (socket says 172.21.15.118:60126) 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.073+0000 7f1a2b7fe700 1 -- 172.21.15.118:0/1304578488 learned_addr learned my addr 172.21.15.118:0/1304578488 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a2b7fe700 1 -- 172.21.15.118:0/1304578488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a2c100ab0 msgr2=0x7f1a2c196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a2b7fe700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a2c100ab0 0x7f1a2c196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a2b7fe700 1 -- 172.21.15.118:0/1304578488 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a200095f0 con 0x7f1a2c101cb0 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a2b7fe700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c196b40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1a200092e0 tx=0x7f1a2000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a297fa700 1 -- 172.21.15.118:0/1304578488 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a2001d070 con 0x7f1a2c101cb0 2024-03-12T03:22:41.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a2c19bc50 con 0x7f1a2c101cb0 2024-03-12T03:22:41.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a297fa700 1 -- 172.21.15.118:0/1304578488 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a2000f460 con 0x7f1a2c101cb0 2024-03-12T03:22:41.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a297fa700 1 -- 172.21.15.118:0/1304578488 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a20021640 con 0x7f1a2c101cb0 2024-03-12T03:22:41.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.074+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a2c19c1c0 con 0x7f1a2c101cb0 2024-03-12T03:22:41.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.075+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2c1906d0 con 0x7f1a2c101cb0 2024-03-12T03:22:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.076+0000 7f1a297fa700 1 -- 172.21.15.118:0/1304578488 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f1a2000f5d0 con 0x7f1a2c101cb0 2024-03-12T03:22:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.076+0000 7f1a297fa700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1a1406c880 0x7f1a1406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.076+0000 7f1a297fa700 1 -- 172.21.15.118:0/1304578488 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f1a2008d000 con 0x7f1a2c101cb0 2024-03-12T03:22:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.076+0000 7f1a2bfff700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1a1406c880 0x7f1a1406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:41.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.076+0000 7f1a2bfff700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1a1406c880 0x7f1a1406ed40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c101b10 tx=0x7f1a1c006d20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:41.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.081+0000 7f1a297fa700 1 -- 172.21.15.118:0/1304578488 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f1a2005b0a0 con 0x7f1a2c101cb0 2024-03-12T03:22:41.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.317+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f1a2c04ea80 con 0x7f1a2c101cb0 2024-03-12T03:22:41.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.318+0000 7f1a297fa700 1 -- 172.21.15.118:0/1304578488 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1114 (secure 0 0 0) 0x7f1a2005ac30 con 0x7f1a2c101cb0 2024-03-12T03:22:41.320 INFO:teuthology.orchestra.run.smithi118.stdout:e2 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:legacy client fscid: 1 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:Filesystem 'cephfs' (1) 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:fs_name cephfs 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 2 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:created 2024-03-12T03:22:37.425947+0000 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:modified 2024-03-12T03:22:37.426018+0000 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:tableserver 0 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:root 0 2024-03-12T03:22:41.321 INFO:teuthology.orchestra.run.smithi118.stdout:session_timeout 60 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:session_autoclose 300 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:max_file_size 1099511627776 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:max_xattr_size 65536 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:required_client_features {} 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:last_failure 0 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:last_failure_osd_epoch 0 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:max_mds 1 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:in 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:up {} 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:failed 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:damaged 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:stopped 2024-03-12T03:22:41.322 INFO:teuthology.orchestra.run.smithi118.stdout:data_pools [3] 2024-03-12T03:22:41.323 INFO:teuthology.orchestra.run.smithi118.stdout:metadata_pool 2 2024-03-12T03:22:41.323 INFO:teuthology.orchestra.run.smithi118.stdout:inline_data disabled 2024-03-12T03:22:41.323 INFO:teuthology.orchestra.run.smithi118.stdout:balancer 2024-03-12T03:22:41.323 INFO:teuthology.orchestra.run.smithi118.stdout:bal_rank_mask -1 2024-03-12T03:22:41.323 INFO:teuthology.orchestra.run.smithi118.stdout:standby_count_wanted 0 2024-03-12T03:22:41.323 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:41.323 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1a1406c880 msgr2=0x7f1a1406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1a1406c880 0x7f1a1406ed40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c101b10 tx=0x7f1a1c006d20 comp rx=0 tx=0).stop 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 msgr2=0x7f1a2c196b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c196b40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1a200092e0 tx=0x7f1a2000be20 comp rx=0 tx=0).stop 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 shutdown_connections 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f1a1406c880 0x7f1a1406ed40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f1a2c100ab0 0x7f1a2c196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 --2- 172.21.15.118:0/1304578488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a2c101cb0 0x7f1a2c196b40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 >> 172.21.15.118:0/1304578488 conn(0x7f1a2c0fc090 msgr2=0x7f1a2c105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 shutdown_connections 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:41.320+0000 7f1a32365700 1 -- 172.21.15.118:0/1304578488 wait complete. 2024-03-12T03:22:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 2 2024-03-12T03:22:41.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:41 smithi177 ceph-mon[37140]: pgmap v180: 65 pgs: 37 active+clean, 9 creating+peering, 19 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:41.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:41 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1304578488' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T03:22:41.987 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:41 smithi118 ceph-mon[29160]: pgmap v180: 65 pgs: 37 active+clean, 9 creating+peering, 19 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:41.987 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:41 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1304578488' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T03:22:42.009 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:22:42.019 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:22:42.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-12T03:22:42.271 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:43.282 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:43.282 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:43.282 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.kbzhid", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.kbzhid", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: Deploying daemon mds.cephfs.smithi177.kbzhid on smithi177 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: pgmap v181: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6826/2160517509,v1:172.21.15.118:6827/2160517509] up:boot 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: daemon mds.cephfs.smithi118.wouiai assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: Cluster is now healthy 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: fsmap cephfs:0 1 up:standby 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.wouiai"}]: dispatch 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: fsmap cephfs:1 {0=cephfs.smithi118.wouiai=up:creating} 2024-03-12T03:22:43.283 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:43 smithi177 ceph-mon[37140]: daemon mds.cephfs.smithi118.wouiai is now active in filesystem cephfs as rank 0 2024-03-12T03:22:43.397 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:43.397 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:43.397 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:43.397 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.kbzhid", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:43.397 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.kbzhid", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:43.397 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: Deploying daemon mds.cephfs.smithi177.kbzhid on smithi177 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: pgmap v181: 65 pgs: 59 active+clean, 6 creating+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6826/2160517509,v1:172.21.15.118:6827/2160517509] up:boot 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: daemon mds.cephfs.smithi118.wouiai assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: Cluster is now healthy 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: fsmap cephfs:0 1 up:standby 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.wouiai"}]: dispatch 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: fsmap cephfs:1 {0=cephfs.smithi118.wouiai=up:creating} 2024-03-12T03:22:43.398 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:43 smithi118 ceph-mon[29160]: daemon mds.cephfs.smithi118.wouiai is now active in filesystem cephfs as rank 0 2024-03-12T03:22:44.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 -- 172.21.15.118:0/2366697258 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0101cb0 msgr2=0x7ff8f0102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:44.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/2366697258 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0101cb0 0x7ff8f0102110 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009910 tx=0x7ff8e0009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 -- 172.21.15.118:0/2366697258 shutdown_connections 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/2366697258 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0101cb0 0x7ff8f0102110 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/2366697258 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0100ab0 0x7ff8f0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 -- 172.21.15.118:0/2366697258 >> 172.21.15.118:0/2366697258 conn(0x7ff8f00fc090 msgr2=0x7ff8f00fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 -- 172.21.15.118:0/2366697258 shutdown_connections 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 -- 172.21.15.118:0/2366697258 wait complete. 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.112+0000 7ff8f7da7700 1 Processor -- start 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f7da7700 1 -- start start 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f7da7700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0100ab0 0x7ff8f0196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f7da7700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0101cb0 0x7ff8f01969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f7da7700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0196ff0 con 0x7ff8f0100ab0 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f7da7700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0197130 con 0x7ff8f0101cb0 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5b43700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0100ab0 0x7ff8f0196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5b43700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0100ab0 0x7ff8f0196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60164/0 (socket says 172.21.15.118:60164) 2024-03-12T03:22:44.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5b43700 1 -- 172.21.15.118:0/3989022045 learned_addr learned my addr 172.21.15.118:0/3989022045 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:44.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5342700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0101cb0 0x7ff8f01969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:44.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5b43700 1 -- 172.21.15.118:0/3989022045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0101cb0 msgr2=0x7ff8f01969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:44.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5b43700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0101cb0 0x7ff8f01969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:44.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5b43700 1 -- 172.21.15.118:0/3989022045 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e00095f0 con 0x7ff8f0100ab0 2024-03-12T03:22:44.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f5b43700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0100ab0 0x7ff8f0196490 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00d900 tx=0x7ff8ec00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:44.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8e6ffd700 1 -- 172.21.15.118:0/3989022045 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8ec00b920 con 0x7ff8f0100ab0 2024-03-12T03:22:44.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f019bb90 con 0x7ff8f0100ab0 2024-03-12T03:22:44.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.113+0000 7ff8e6ffd700 1 -- 172.21.15.118:0/3989022045 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8ec010460 con 0x7ff8f0100ab0 2024-03-12T03:22:44.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.114+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f019c0e0 con 0x7ff8f0100ab0 2024-03-12T03:22:44.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.114+0000 7ff8e6ffd700 1 -- 172.21.15.118:0/3989022045 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff8ec00f590 con 0x7ff8f0100ab0 2024-03-12T03:22:44.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.114+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f0190630 con 0x7ff8f0100ab0 2024-03-12T03:22:44.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.115+0000 7ff8e6ffd700 1 -- 172.21.15.118:0/3989022045 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff8ec0105d0 con 0x7ff8f0100ab0 2024-03-12T03:22:44.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.115+0000 7ff8e6ffd700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff8dc06c890 0x7ff8dc06ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:44.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.116+0000 7ff8f5342700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff8dc06c890 0x7ff8dc06ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:44.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.116+0000 7ff8e6ffd700 1 -- 172.21.15.118:0/3989022045 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7ff8ec08bc30 con 0x7ff8f0100ab0 2024-03-12T03:22:44.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.116+0000 7ff8f5342700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff8dc06c890 0x7ff8dc06ed50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0006010 tx=0x7ff8e0005180 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:44.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.120+0000 7ff8e6ffd700 1 -- 172.21.15.118:0/3989022045 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff8ec059cd0 con 0x7ff8f0100ab0 2024-03-12T03:22:44.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.356+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7ff8f00619c0 con 0x7ff8f0100ab0 2024-03-12T03:22:44.772 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:44 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6826/2160517509,v1:172.21.15.118:6827/2160517509] up:active 2024-03-12T03:22:44.772 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:44 smithi177 ceph-mon[37140]: fsmap cephfs:1 {0=cephfs.smithi118.wouiai=up:active} 2024-03-12T03:22:44.772 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:44 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3989022045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-12T03:22:44.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.905+0000 7ff8e6ffd700 1 -- 172.21.15.118:0/3989022045 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff8ec059860 con 0x7ff8f0100ab0 2024-03-12T03:22:44.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.907+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff8dc06c890 msgr2=0x7ff8dc06ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:44.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.907+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff8dc06c890 0x7ff8dc06ed50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0006010 tx=0x7ff8e0005180 comp rx=0 tx=0).stop 2024-03-12T03:22:44.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.907+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0100ab0 msgr2=0x7ff8f0196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:44.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.907+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0100ab0 0x7ff8f0196490 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec00d900 tx=0x7ff8ec00dc10 comp rx=0 tx=0).stop 2024-03-12T03:22:44.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.908+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 shutdown_connections 2024-03-12T03:22:44.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.908+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff8dc06c890 0x7ff8dc06ed50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:44.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.908+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8f0100ab0 0x7ff8f0196490 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:44.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.908+0000 7ff8f7da7700 1 --2- 172.21.15.118:0/3989022045 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff8f0101cb0 0x7ff8f01969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:44.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.908+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 >> 172.21.15.118:0/3989022045 conn(0x7ff8f00fc090 msgr2=0x7ff8f0105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:44.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.908+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 shutdown_connections 2024-03-12T03:22:44.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:44.908+0000 7ff8f7da7700 1 -- 172.21.15.118:0/3989022045 wait complete. 2024-03-12T03:22:44.928 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:44 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6826/2160517509,v1:172.21.15.118:6827/2160517509] up:active 2024-03-12T03:22:44.928 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:44 smithi118 ceph-mon[29160]: fsmap cephfs:1 {0=cephfs.smithi118.wouiai=up:active} 2024-03-12T03:22:44.928 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:44 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3989022045' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-12T03:22:45.554 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:22:45.564 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:22:45.564 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: pgmap v182: 65 pgs: 65 active+clean; 578 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 861 B/s wr, 5 op/s 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.oysrod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3989022045' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-12T03:22:45.815 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.177:6824/4286002121,v1:172.21.15.177:6825/4286002121] up:boot 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: daemon mds.cephfs.smithi177.kbzhid assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: fsmap cephfs:1 {0=cephfs.smithi118.wouiai=up:active} 1 up:standby 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.kbzhid"}]: dispatch 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.oysrod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:creating} 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:45.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:45 smithi118 ceph-mon[29160]: daemon mds.cephfs.smithi177.kbzhid is now active in filesystem cephfs as rank 1 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: pgmap v182: 65 pgs: 65 active+clean; 578 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 861 B/s wr, 5 op/s 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.oysrod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3989022045' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.177:6824/4286002121,v1:172.21.15.177:6825/4286002121] up:boot 2024-03-12T03:22:45.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: daemon mds.cephfs.smithi177.kbzhid assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: fsmap cephfs:1 {0=cephfs.smithi118.wouiai=up:active} 1 up:standby 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.kbzhid"}]: dispatch 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi118.oysrod", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:creating} 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:45.954 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:45 smithi177 ceph-mon[37140]: daemon mds.cephfs.smithi177.kbzhid is now active in filesystem cephfs as rank 1 2024-03-12T03:22:46.297 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:46.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:46 smithi177 ceph-mon[37140]: Deploying daemon mds.cephfs.smithi118.oysrod on smithi118 2024-03-12T03:22:46.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:46 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.177:6824/4286002121,v1:172.21.15.177:6825/4286002121] up:active 2024-03-12T03:22:46.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:46 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 2024-03-12T03:22:47.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:46 smithi118 ceph-mon[29160]: Deploying daemon mds.cephfs.smithi118.oysrod on smithi118 2024-03-12T03:22:47.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:46 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.177:6824/4286002121,v1:172.21.15.177:6825/4286002121] up:active 2024-03-12T03:22:47.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:46 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 2024-03-12T03:22:47.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:47 smithi118 ceph-mon[29160]: pgmap v183: 65 pgs: 65 active+clean; 578 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-12T03:22:47.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:47 smithi177 ceph-mon[37140]: pgmap v183: 65 pgs: 65 active+clean; 578 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-12T03:22:48.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 -- 172.21.15.118:0/1373396470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 msgr2=0x7f5ac40ffdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:48.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 --2- 172.21.15.118:0/1373396470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac40ffdc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4009910 tx=0x7f5ab4009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:48.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 -- 172.21.15.118:0/1373396470 shutdown_connections 2024-03-12T03:22:48.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 --2- 172.21.15.118:0/1373396470 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ac4100ba0 0x7f5ac4101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 --2- 172.21.15.118:0/1373396470 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac40ffdc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 -- 172.21.15.118:0/1373396470 >> 172.21.15.118:0/1373396470 conn(0x7f5ac40faf40 msgr2=0x7f5ac40fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 -- 172.21.15.118:0/1373396470 shutdown_connections 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 -- 172.21.15.118:0/1373396470 wait complete. 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 Processor -- start 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.337+0000 7f5acaa44700 1 -- start start 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5acaa44700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac41038e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5acaa44700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ac4100ba0 0x7f5ac4101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5acaa44700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac41024c0 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5acaa44700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4102600 con 0x7f5ac4100ba0 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac41038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac41038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35122/0 (socket says 172.21.15.118:35122) 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac3fff700 1 -- 172.21.15.118:0/4137415082 learned_addr learned my addr 172.21.15.118:0/4137415082 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:48.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac37fe700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ac4100ba0 0x7f5ac4101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac3fff700 1 -- 172.21.15.118:0/4137415082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ac4100ba0 msgr2=0x7f5ac4101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac3fff700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ac4100ba0 0x7f5ac4101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac3fff700 1 -- 172.21.15.118:0/4137415082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab40095f0 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.338+0000 7f5ac3fff700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac41038e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40059d0 tx=0x7f5ab400bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.339+0000 7f5ac17fa700 1 -- 172.21.15.118:0/4137415082 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab401d070 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.339+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac4102880 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.339+0000 7f5ac17fa700 1 -- 172.21.15.118:0/4137415082 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ab4005dc0 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.339+0000 7f5ac17fa700 1 -- 172.21.15.118:0/4137415082 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab4021410 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.339+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac4102df0 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.339+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac404ea80 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.340+0000 7f5ac17fa700 1 -- 172.21.15.118:0/4137415082 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f5ab400f460 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.340+0000 7f5ac17fa700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f5aac06c880 0x7f5aac06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:48.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.341+0000 7f5ac17fa700 1 -- 172.21.15.118:0/4137415082 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7f5ab408ccb0 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.341+0000 7f5ac37fe700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f5aac06c880 0x7f5aac06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:48.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.341+0000 7f5ac37fe700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f5aac06c880 0x7f5aac06ed40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8005950 tx=0x7f5ab800b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:48.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.345+0000 7f5ac17fa700 1 -- 172.21.15.118:0/4137415082 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5ab405ad50 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.596+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f5ac4102df0 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.901+0000 7f5ac17fa700 1 -- 172.21.15.118:0/4137415082 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f5ab405a8e0 con 0x7f5ac40ff9a0 2024-03-12T03:22:48.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f5aac06c880 msgr2=0x7f5aac06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:48.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f5aac06c880 0x7f5aac06ed40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8005950 tx=0x7f5ab800b410 comp rx=0 tx=0).stop 2024-03-12T03:22:48.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 msgr2=0x7f5ac41038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:48.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac41038e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5ab40059d0 tx=0x7f5ab400bd70 comp rx=0 tx=0).stop 2024-03-12T03:22:48.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 shutdown_connections 2024-03-12T03:22:48.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f5aac06c880 0x7f5aac06ed40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:48.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ac40ff9a0 0x7f5ac41038e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:48.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 --2- 172.21.15.118:0/4137415082 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f5ac4100ba0 0x7f5ac4101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:48.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 >> 172.21.15.118:0/4137415082 conn(0x7f5ac40faf40 msgr2=0x7f5ac4069040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:48.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 shutdown_connections 2024-03-12T03:22:48.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:48.905+0000 7f5acaa44700 1 -- 172.21.15.118:0/4137415082 wait complete. 2024-03-12T03:22:49.562 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:22:49.572 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:22:49.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-03-12T03:22:49.783 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:49.808 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-03-12T03:22:49.808 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.808 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:49.808 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.808 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/4137415082' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T03:22:49.808 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-12T03:22:49.808 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: Cluster is now healthy 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/4137415082' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:boot 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.oysrod"}]: dispatch 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.azybll", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.azybll", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:49.809 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:49 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s wr, 7 op/s 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/4137415082' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: Cluster is now healthy 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/4137415082' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:boot 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.oysrod"}]: dispatch 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:49.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.azybll", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-12T03:22:49.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi177.azybll", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-12T03:22:49.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:49 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:22:50.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:50 smithi177 ceph-mon[37140]: Deploying daemon mds.cephfs.smithi177.azybll on smithi177 2024-03-12T03:22:50.806 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:50 smithi118 ceph-mon[29160]: Deploying daemon mds.cephfs.smithi177.azybll on smithi177 2024-03-12T03:22:51.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 -- 172.21.15.118:0/2842833011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 msgr2=0x7fede0100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:51.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 --2- 172.21.15.118:0/2842833011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede0100fe0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009910 tx=0x7fedd0009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:51.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 -- 172.21.15.118:0/2842833011 shutdown_connections 2024-03-12T03:22:51.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 --2- 172.21.15.118:0/2842833011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede0100fe0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:51.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 --2- 172.21.15.118:0/2842833011 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede00ff980 0x7fede00ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:51.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 -- 172.21.15.118:0/2842833011 >> 172.21.15.118:0/2842833011 conn(0x7fede00faf60 msgr2=0x7fede00fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 -- 172.21.15.118:0/2842833011 shutdown_connections 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.321+0000 7fede8395700 1 -- 172.21.15.118:0/2842833011 wait complete. 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede8395700 1 Processor -- start 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede8395700 1 -- start start 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede8395700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede00ff980 0x7fede0068f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede8395700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede019b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede8395700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede019b780 con 0x7fede0100b80 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede8395700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede019b8c0 con 0x7fede00ff980 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede5930700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede019b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede5930700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede019b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35160/0 (socket says 172.21.15.118:35160) 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede5930700 1 -- 172.21.15.118:0/3580452717 learned_addr learned my addr 172.21.15.118:0/3580452717 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede6131700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede00ff980 0x7fede0068f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede5930700 1 -- 172.21.15.118:0/3580452717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede00ff980 msgr2=0x7fede0068f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede5930700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede00ff980 0x7fede0068f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:51.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede5930700 1 -- 172.21.15.118:0/3580452717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd00095f0 con 0x7fede0100b80 2024-03-12T03:22:51.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.322+0000 7fede5930700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede019b1b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fedd000b800 tx=0x7fedd0005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:51.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.323+0000 7fedd77fe700 1 -- 172.21.15.118:0/3580452717 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd001d070 con 0x7fede0100b80 2024-03-12T03:22:51.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.323+0000 7fedd77fe700 1 -- 172.21.15.118:0/3580452717 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fedd00052a0 con 0x7fede0100b80 2024-03-12T03:22:51.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.323+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede019baf0 con 0x7fede0100b80 2024-03-12T03:22:51.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.323+0000 7fedd77fe700 1 -- 172.21.15.118:0/3580452717 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fedd0017400 con 0x7fede0100b80 2024-03-12T03:22:51.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.323+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede019c060 con 0x7fede0100b80 2024-03-12T03:22:51.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.324+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede018f500 con 0x7fede0100b80 2024-03-12T03:22:51.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.324+0000 7fedd77fe700 1 -- 172.21.15.118:0/3580452717 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fedd0003780 con 0x7fede0100b80 2024-03-12T03:22:51.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.324+0000 7fedd77fe700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fedcc06c880 0x7fedcc06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:51.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.325+0000 7fede6131700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fedcc06c880 0x7fedcc06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:51.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.325+0000 7fedd77fe700 1 -- 172.21.15.118:0/3580452717 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5468+0+0 (secure 0 0 0) 0x7fedd008d0f0 con 0x7fede0100b80 2024-03-12T03:22:51.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.325+0000 7fede6131700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fedcc06c880 0x7fedcc06ed40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fede01009e0 tx=0x7feddc009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:51.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.329+0000 7fedd77fe700 1 -- 172.21.15.118:0/3580452717 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fedd005b190 con 0x7fede0100b80 2024-03-12T03:22:51.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.567+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fede002cf30 con 0x7fede0100b80 2024-03-12T03:22:51.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.593+0000 7fedd77fe700 1 -- 172.21.15.118:0/3580452717 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v10) v1 ==== 168+0+0 (secure 0 0 0) 0x7fedd005ad20 con 0x7fede0100b80 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fedcc06c880 msgr2=0x7fedcc06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fedcc06c880 0x7fedcc06ed40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fede01009e0 tx=0x7feddc009260 comp rx=0 tx=0).stop 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 msgr2=0x7fede019b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede019b1b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fedd000b800 tx=0x7fedd0005f00 comp rx=0 tx=0).stop 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 shutdown_connections 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fedcc06c880 0x7fedcc06ed40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fede00ff980 0x7fede0068f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:51.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 --2- 172.21.15.118:0/3580452717 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fede0100b80 0x7fede019b1b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:51.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 >> 172.21.15.118:0/3580452717 conn(0x7fede00faf60 msgr2=0x7fede01041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:51.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.595+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 shutdown_connections 2024-03-12T03:22:51.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:51.596+0000 7fede8395700 1 -- 172.21.15.118:0/3580452717 wait complete. 2024-03-12T03:22:51.599 INFO:teuthology.orchestra.run.smithi118.stderr:inline data enabled 2024-03-12T03:22:51.939 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:51 smithi118 ceph-mon[29160]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 91 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-12T03:22:51.940 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:51 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3580452717' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:22:51.940 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:51 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:51.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:51 smithi177 ceph-mon[37140]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 91 B/s rd, 2.3 KiB/s wr, 7 op/s 2024-03-12T03:22:51.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:51 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3580452717' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-12T03:22:51.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:51 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.210 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:22:52.222 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:22:52.222 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-12T03:22:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-12T03:22:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.177:6826/1551126945,v1:172.21.15.177:6827/1551126945] up:boot 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3580452717' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.azybll"}]: dispatch 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 2 up:standby-replay 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:52 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:52.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.177:6826/1551126945,v1:172.21.15.177:6827/1551126945] up:boot 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3580452717' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.azybll"}]: dispatch 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 2 up:standby-replay 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:52.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:52 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:22:53.078 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:53.808 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:53 smithi118 ceph-mon[29160]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-12T03:22:53.808 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:53.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:53 smithi177 ceph-mon[37140]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 255 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-12T03:22:53.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:54.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 -- 172.21.15.118:0/1060330405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 msgr2=0x7f85b40fef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:54.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 --2- 172.21.15.118:0/1060330405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b40fef90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009910 tx=0x7f85a8009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:54.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 -- 172.21.15.118:0/1060330405 shutdown_connections 2024-03-12T03:22:54.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 --2- 172.21.15.118:0/1060330405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b40fef90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 --2- 172.21.15.118:0/1060330405 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b41020b0 0x7f85b41024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 -- 172.21.15.118:0/1060330405 >> 172.21.15.118:0/1060330405 conn(0x7f85b40fa6d0 msgr2=0x7f85b40fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 -- 172.21.15.118:0/1060330405 shutdown_connections 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.636+0000 7f85baa13700 1 -- 172.21.15.118:0/1060330405 wait complete. 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85baa13700 1 Processor -- start 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85baa13700 1 -- start start 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85baa13700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b4196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85baa13700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b41020b0 0x7f85b4196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85baa13700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b41970a0 con 0x7f85b40feb30 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85baa13700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b41971e0 con 0x7f85b41020b0 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85b3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b4196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85b3fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b4196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42898/0 (socket says 172.21.15.118:42898) 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85b3fff700 1 -- 172.21.15.118:0/4276835077 learned_addr learned my addr 172.21.15.118:0/4276835077 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:54.641 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.637+0000 7f85b37fe700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b41020b0 0x7f85b4196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b3fff700 1 -- 172.21.15.118:0/4276835077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b41020b0 msgr2=0x7f85b4196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b3fff700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b41020b0 0x7f85b4196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b3fff700 1 -- 172.21.15.118:0/4276835077 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a80095f0 con 0x7f85b40feb30 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b37fe700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b41020b0 0x7f85b4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b3fff700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b4196540 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85a400e8f0 tx=0x7f85a400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b17fa700 1 -- 172.21.15.118:0/4276835077 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85a4009790 con 0x7f85b40feb30 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b419bbf0 con 0x7f85b40feb30 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b17fa700 1 -- 172.21.15.118:0/4276835077 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85a40044e0 con 0x7f85b40feb30 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85b17fa700 1 -- 172.21.15.118:0/4276835077 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85a4003e00 con 0x7f85b40feb30 2024-03-12T03:22:54.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.638+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b419c140 con 0x7f85b40feb30 2024-03-12T03:22:54.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.639+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b4190660 con 0x7f85b40feb30 2024-03-12T03:22:54.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.639+0000 7f85b17fa700 1 -- 172.21.15.118:0/4276835077 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f85a4004020 con 0x7f85b40feb30 2024-03-12T03:22:54.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.640+0000 7f85b17fa700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f859c06c820 0x7f859c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:54.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.640+0000 7f85b37fe700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f859c06c820 0x7f859c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:54.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.640+0000 7f85b17fa700 1 -- 172.21.15.118:0/4276835077 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f85a4014070 con 0x7f85b40feb30 2024-03-12T03:22:54.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.640+0000 7f85b37fe700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f859c06c820 0x7f859c06ece0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85a8006010 tx=0x7f85a8009f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:54.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.644+0000 7f85b17fa700 1 -- 172.21.15.118:0/4276835077 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f85a405a100 con 0x7f85b40feb30 2024-03-12T03:22:54.782 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:standby-replay 2024-03-12T03:22:54.782 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.177:6824/4286002121,v1:172.21.15.177:6825/4286002121] up:active 2024-03-12T03:22:54.782 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: Dropping low affinity active daemon mds.cephfs.smithi118.wouiai in favor of higher affinity standby. 2024-03-12T03:22:54.782 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: Replacing daemon mds.cephfs.smithi118.wouiai as rank 0 with standby daemon mds.cephfs.smithi118.oysrod 2024-03-12T03:22:54.782 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T03:22:54.783 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 2 up:standby-replay 2024-03-12T03:22:54.783 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: osdmap e47: 6 total, 6 up, 6 in 2024-03-12T03:22:54.783 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:replay,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 2024-03-12T03:22:54.783 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:54.783 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:54 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:54.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.894+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f85b402ccb0 con 0x7f85b40feb30 2024-03-12T03:22:54.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.895+0000 7f85b17fa700 1 -- 172.21.15.118:0/4276835077 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 14 v14) v1 ==== 76+0+1874 (secure 0 0 0) 0x7f85a4059c90 con 0x7f85b40feb30 2024-03-12T03:22:54.897 INFO:teuthology.orchestra.run.smithi118.stdout:e14 2024-03-12T03:22:54.897 INFO:teuthology.orchestra.run.smithi118.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:22:54.897 INFO:teuthology.orchestra.run.smithi118.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:22:54.897 INFO:teuthology.orchestra.run.smithi118.stdout:legacy client fscid: 1 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:Filesystem 'cephfs' (1) 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:fs_name cephfs 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 14 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:created 2024-03-12T03:22:37.425947+0000 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:modified 2024-03-12T03:22:54.630049+0000 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:tableserver 0 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:root 0 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:session_timeout 60 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:session_autoclose 300 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:max_file_size 1099511627776 2024-03-12T03:22:54.898 INFO:teuthology.orchestra.run.smithi118.stdout:max_xattr_size 65536 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:required_client_features {} 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:last_failure 0 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:last_failure_osd_epoch 47 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:max_mds 2 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:in 0,1 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:up {0=14486,1=24279} 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:failed 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:damaged 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:stopped 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:data_pools [3] 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:metadata_pool 2 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:inline_data enabled 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:balancer 2024-03-12T03:22:54.899 INFO:teuthology.orchestra.run.smithi118.stdout:bal_rank_mask -1 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout:standby_count_wanted 1 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.cephfs.smithi118.oysrod{0:14486} state up:resolve seq 3 join_fscid=1 addr [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] compat {c=[1],r=[1],i=[7ff]}] 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.cephfs.smithi177.kbzhid{1:24279} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.177:6824/4286002121,v1:172.21.15.177:6825/4286002121] compat {c=[1],r=[1],i=[7ff]}] 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.cephfs.smithi177.azybll{1:24291} state up:standby-replay seq 1 addr [v2:172.21.15.177:6826/1551126945,v1:172.21.15.177:6827/1551126945] compat {c=[1],r=[1],i=[7ff]}] 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout:Standby daemons: 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:22:54.900 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.cephfs.smithi118.wouiai{-1:14498} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.118:6826/931262818,v1:172.21.15.118:6827/931262818] compat {c=[1],r=[1],i=[7ff]}] 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f859c06c820 msgr2=0x7f859c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f859c06c820 0x7f859c06ece0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f85a8006010 tx=0x7f85a8009f90 comp rx=0 tx=0).stop 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 msgr2=0x7f85b4196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b4196540 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f85a400e8f0 tx=0x7f85a400ecb0 comp rx=0 tx=0).stop 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 shutdown_connections 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f859c06c820 0x7f859c06ece0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f85b40feb30 0x7f85b4196540 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 --2- 172.21.15.118:0/4276835077 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f85b41020b0 0x7f85b4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 >> 172.21.15.118:0/4276835077 conn(0x7f85b40fa6d0 msgr2=0x7f85b4105450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:54.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 shutdown_connections 2024-03-12T03:22:54.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:54.897+0000 7f85baa13700 1 -- 172.21.15.118:0/4276835077 wait complete. 2024-03-12T03:22:54.902 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 14 2024-03-12T03:22:54.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:standby-replay 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.177:6824/4286002121,v1:172.21.15.177:6825/4286002121] up:active 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: Dropping low affinity active daemon mds.cephfs.smithi118.wouiai in favor of higher affinity standby. 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: Replacing daemon mds.cephfs.smithi118.wouiai as rank 0 with standby daemon mds.cephfs.smithi118.oysrod 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.wouiai=up:active,1=cephfs.smithi177.kbzhid=up:active} 2 up:standby-replay 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: osdmap e47: 6 total, 6 up, 6 in 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:replay,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:54.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:54 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:22:55.609 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-12T03:22:55.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:55 smithi177 ceph-mon[37140]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-12T03:22:55.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:55 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6826/931262818,v1:172.21.15.118:6827/931262818] up:boot 2024-03-12T03:22:55.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:55 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:resolve 2024-03-12T03:22:55.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:55 smithi177 ceph-mon[37140]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:resolve,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:55.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:55 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.wouiai"}]: dispatch 2024-03-12T03:22:55.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:55 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/4276835077' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T03:22:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:55 smithi118 ceph-mon[29160]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-12T03:22:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:55 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6826/931262818,v1:172.21.15.118:6827/931262818] up:boot 2024-03-12T03:22:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:55 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:resolve 2024-03-12T03:22:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:55 smithi118 ceph-mon[29160]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:resolve,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:56.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:55 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.wouiai"}]: dispatch 2024-03-12T03:22:56.062 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:55 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/4276835077' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-12T03:22:56.651 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:22:56.931 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:56 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:reconnect 2024-03-12T03:22:56.931 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:56 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.177:6826/1551126945,v1:172.21.15.177:6827/1551126945] up:standby-replay 2024-03-12T03:22:56.931 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:56 smithi177 ceph-mon[37140]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:reconnect,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:56.985 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:56 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:reconnect 2024-03-12T03:22:56.985 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:56 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.177:6826/1551126945,v1:172.21.15.177:6827/1551126945] up:standby-replay 2024-03-12T03:22:56.985 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:56 smithi118 ceph-mon[29160]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:reconnect,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:57 smithi177 ceph-mon[37140]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-12T03:22:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:57 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:rejoin 2024-03-12T03:22:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:57 smithi177 ceph-mon[37140]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:rejoin,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:57 smithi177 ceph-mon[37140]: daemon mds.cephfs.smithi118.oysrod is now active in filesystem cephfs as rank 0 2024-03-12T03:22:57.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:57 smithi118 ceph-mon[29160]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-03-12T03:22:57.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:57 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:rejoin 2024-03-12T03:22:57.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:57 smithi118 ceph-mon[29160]: fsmap cephfs:2/2 {0=cephfs.smithi118.oysrod=up:rejoin,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:57.972 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:57 smithi118 ceph-mon[29160]: daemon mds.cephfs.smithi118.oysrod is now active in filesystem cephfs as rank 0 2024-03-12T03:22:58.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 -- 172.21.15.118:0/1637733027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40100a90 msgr2=0x7fcb40100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:58.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 --2- 172.21.15.118:0/1637733027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40100a90 0x7fcb40100eb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009910 tx=0x7fcb30009c20 comp rx=0 tx=0).stop 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 -- 172.21.15.118:0/1637733027 shutdown_connections 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 --2- 172.21.15.118:0/1637733027 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb40101c90 0x7fcb401020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 --2- 172.21.15.118:0/1637733027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40100a90 0x7fcb40100eb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 -- 172.21.15.118:0/1637733027 >> 172.21.15.118:0/1637733027 conn(0x7fcb400fc050 msgr2=0x7fcb400fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 -- 172.21.15.118:0/1637733027 shutdown_connections 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.576+0000 7fcb485ea700 1 -- 172.21.15.118:0/1637733027 wait complete. 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb485ea700 1 Processor -- start 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb485ea700 1 -- start start 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb485ea700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb40100a90 0x7fcb401965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb485ea700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40101c90 0x7fcb40196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb485ea700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb40197130 con 0x7fcb40101c90 2024-03-12T03:22:58.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb485ea700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb40197270 con 0x7fcb40100a90 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb45b85700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40101c90 0x7fcb40196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb45b85700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40101c90 0x7fcb40196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42924/0 (socket says 172.21.15.118:42924) 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb45b85700 1 -- 172.21.15.118:0/184226554 learned_addr learned my addr 172.21.15.118:0/184226554 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb46386700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb40100a90 0x7fcb401965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.577+0000 7fcb45b85700 1 -- 172.21.15.118:0/184226554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb40100a90 msgr2=0x7fcb401965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb45b85700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb40100a90 0x7fcb401965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb45b85700 1 -- 172.21.15.118:0/184226554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb300095f0 con 0x7fcb40101c90 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb45b85700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40101c90 0x7fcb40196b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00b4b0 tx=0x7fcb3c00b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb377fe700 1 -- 172.21.15.118:0/184226554 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c011800 con 0x7fcb40101c90 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb377fe700 1 -- 172.21.15.118:0/184226554 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb3c011e20 con 0x7fcb40101c90 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4019bc80 con 0x7fcb40101c90 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb377fe700 1 -- 172.21.15.118:0/184226554 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb3c00f2e0 con 0x7fcb40101c90 2024-03-12T03:22:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.578+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4019c250 con 0x7fcb40101c90 2024-03-12T03:22:58.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.579+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb40190690 con 0x7fcb40101c90 2024-03-12T03:22:58.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.579+0000 7fcb377fe700 1 -- 172.21.15.118:0/184226554 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcb3c010380 con 0x7fcb40101c90 2024-03-12T03:22:58.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.580+0000 7fcb377fe700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb2c06c880 0x7fcb2c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:22:58.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.580+0000 7fcb377fe700 1 -- 172.21.15.118:0/184226554 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcb3c08b850 con 0x7fcb40101c90 2024-03-12T03:22:58.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.580+0000 7fcb46386700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb2c06c880 0x7fcb2c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:22:58.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.580+0000 7fcb46386700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb2c06c880 0x7fcb2c06ed40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcb30000c00 tx=0x7fcb30005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:22:58.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.584+0000 7fcb377fe700 1 -- 172.21.15.118:0/184226554 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcb3c059890 con 0x7fcb40101c90 2024-03-12T03:22:58.668 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:58 smithi118 ceph-mon[29160]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T03:22:58.668 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:58 smithi118 ceph-mon[29160]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:active 2024-03-12T03:22:58.668 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:58 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.oysrod=up:active,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:58.668 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:58 smithi118 ceph-mon[29160]: fsmap cephfs:2 {0=cephfs.smithi118.oysrod=up:active,1=cephfs.smithi177.kbzhid=up:active} 2 up:standby-replay 2024-03-12T03:22:58.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.835+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb400623e0 con 0x7fcb40101c90 2024-03-12T03:22:58.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.836+0000 7fcb377fe700 1 -- 172.21.15.118:0/184226554 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4865 (secure 0 0 0) 0x7fcb3c059420 con 0x7fcb40101c90 2024-03-12T03:22:58.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb2c06c880 msgr2=0x7fcb2c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:58.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb2c06c880 0x7fcb2c06ed40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcb30000c00 tx=0x7fcb30005960 comp rx=0 tx=0).stop 2024-03-12T03:22:58.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40101c90 msgr2=0x7fcb40196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:22:58.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40101c90 0x7fcb40196b10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c00b4b0 tx=0x7fcb3c00b7c0 comp rx=0 tx=0).stop 2024-03-12T03:22:58.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 shutdown_connections 2024-03-12T03:22:58.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb2c06c880 0x7fcb2c06ed40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:58.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcb40100a90 0x7fcb401965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 --2- 172.21.15.118:0/184226554 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcb40101c90 0x7fcb40196b10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:22:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 >> 172.21.15.118:0/184226554 conn(0x7fcb400fc050 msgr2=0x7fcb401052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:22:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 shutdown_connections 2024-03-12T03:22:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:22:58.838+0000 7fcb485ea700 1 -- 172.21.15.118:0/184226554 wait complete. 2024-03-12T03:22:58.841 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 18 2024-03-12T03:22:58.850 INFO:teuthology.orchestra.run.smithi118.stdout:true 2024-03-12T03:22:58.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:58 smithi177 ceph-mon[37140]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-12T03:22:58.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:58 smithi177 ceph-mon[37140]: mds.? [v2:172.21.15.118:6828/2437365950,v1:172.21.15.118:6829/2437365950] up:active 2024-03-12T03:22:58.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:58 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.oysrod=up:active,1=cephfs.smithi177.kbzhid=up:active} 1 up:standby-replay 1 up:standby 2024-03-12T03:22:58.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:58 smithi177 ceph-mon[37140]: fsmap cephfs:2 {0=cephfs.smithi118.oysrod=up:active,1=cephfs.smithi177.kbzhid=up:active} 2 up:standby-replay 2024-03-12T03:22:59.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-12T03:22:59.785 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:59 smithi118 ceph-mon[29160]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 5.9 KiB/s rd, 409 B/s wr, 8 op/s 2024-03-12T03:22:59.785 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:22:59 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/184226554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T03:22:59.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:59 smithi177 ceph-mon[37140]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 5.9 KiB/s rd, 409 B/s wr, 8 op/s 2024-03-12T03:22:59.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:22:59 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/184226554' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T03:23:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:01.754 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:01 smithi118 ceph-mon[29160]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 9.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 -- 172.21.15.118:0/646482599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 msgr2=0x7f9e100fefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 --2- 172.21.15.118:0/646482599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e100fefa0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009910 tx=0x7f9e04009c20 comp rx=0 tx=0).stop 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 -- 172.21.15.118:0/646482599 shutdown_connections 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 --2- 172.21.15.118:0/646482599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e100fefa0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 --2- 172.21.15.118:0/646482599 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e101020c0 0x7f9e101024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 -- 172.21.15.118:0/646482599 >> 172.21.15.118:0/646482599 conn(0x7f9e100fa700 msgr2=0x7f9e100fcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 -- 172.21.15.118:0/646482599 shutdown_connections 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.853+0000 7f9e16505700 1 -- 172.21.15.118:0/646482599 wait complete. 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e16505700 1 Processor -- start 2024-03-12T03:23:01.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e16505700 1 -- start start 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e16505700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e101966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e16505700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e101020c0 0x7f9e10196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e16505700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10197230 con 0x7f9e100feb40 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e16505700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10197370 con 0x7f9e101020c0 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e0ffff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e101966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e0ffff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e101966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42950/0 (socket says 172.21.15.118:42950) 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.854+0000 7f9e0ffff700 1 -- 172.21.15.118:0/475968729 learned_addr learned my addr 172.21.15.118:0/475968729 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0f7fe700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e101020c0 0x7f9e10196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0ffff700 1 -- 172.21.15.118:0/475968729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e101020c0 msgr2=0x7f9e10196c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0ffff700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e101020c0 0x7f9e10196c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:01.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0ffff700 1 -- 172.21.15.118:0/475968729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e040095f0 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0ffff700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e101966d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000b480 tx=0x7f9e0000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0d7fa700 1 -- 172.21.15.118:0/475968729 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e000118c0 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1019bd80 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0d7fa700 1 -- 172.21.15.118:0/475968729 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e00010380 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e0d7fa700 1 -- 172.21.15.118:0/475968729 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9e0000f4f0 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.855+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1019c2f0 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.856+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1004eaf0 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.857+0000 7f9e0d7fa700 1 -- 172.21.15.118:0/475968729 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f9e000104f0 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.857+0000 7f9e0d7fa700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9df806c890 0x7f9df806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.857+0000 7f9e0d7fa700 1 -- 172.21.15.118:0/475968729 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9e0008b960 con 0x7f9e100feb40 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.857+0000 7f9e0f7fe700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9df806c890 0x7f9df806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:01.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.857+0000 7f9e0f7fe700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9df806c890 0x7f9df806ed50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400b5c0 tx=0x7f9e04005ee0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:01.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:01.862+0000 7f9e0d7fa700 1 -- 172.21.15.118:0/475968729 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9e1004eaf0 con 0x7f9e100feb40 2024-03-12T03:23:01.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:01 smithi177 ceph-mon[37140]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 9.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:23:02.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.152+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f9e1004ea80 con 0x7f9e100feb40 2024-03-12T03:23:02.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.152+0000 7f9e0d7fa700 1 -- 172.21.15.118:0/475968729 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v18) v1 ==== 78+0+97 (secure 0 0 0) 0x7f9e000599a0 con 0x7f9e100feb40 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.154+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9df806c890 msgr2=0x7f9df806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.154+0000 7f9e16505700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9df806c890 0x7f9df806ed50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9e0400b5c0 tx=0x7f9e04005ee0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.154+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 msgr2=0x7f9e101966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.154+0000 7f9e16505700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e101966d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000b480 tx=0x7f9e0000b840 comp rx=0 tx=0).stop 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.155+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 shutdown_connections 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.155+0000 7f9e16505700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f9df806c890 0x7f9df806ed50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.155+0000 7f9e16505700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e100feb40 0x7f9e101966d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.155+0000 7f9e16505700 1 --2- 172.21.15.118:0/475968729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f9e101020c0 0x7f9e10196c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.155+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 >> 172.21.15.118:0/475968729 conn(0x7f9e100fa700 msgr2=0x7f9e101055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:02.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.155+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 shutdown_connections 2024-03-12T03:23:02.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.155+0000 7f9e16505700 1 -- 172.21.15.118:0/475968729 wait complete. 2024-03-12T03:23:02.168 INFO:teuthology.orchestra.run.smithi118.stdout:true 2024-03-12T03:23:02.746 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-12T03:23:02.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 -- 172.21.15.118:0/3733431147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8240fffe0 msgr2=0x7fc824100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 --2- 172.21.15.118:0/3733431147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8240fffe0 0x7fc824100460 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc810009880 tx=0x7fc81002bce0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 -- 172.21.15.118:0/3733431147 shutdown_connections 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 --2- 172.21.15.118:0/3733431147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8240fffe0 0x7fc824100460 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc810009880 tx=0x7fc81002bce0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 --2- 172.21.15.118:0/3733431147 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8240fee90 0x7fc8240ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 -- 172.21.15.118:0/3733431147 >> 172.21.15.118:0/3733431147 conn(0x7fc8240fa440 msgr2=0x7fc8240fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 -- 172.21.15.118:0/3733431147 shutdown_connections 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 -- 172.21.15.118:0/3733431147 wait complete. 2024-03-12T03:23:02.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 Processor -- start 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.877+0000 7fc82af1b640 1 -- start start 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc82af1b640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8240fee90 0x7fc824194c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc82af1b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc824195150 0x7fc82419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc82af1b640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8241955d0 con 0x7fc824195150 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc82af1b640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc824195740 con 0x7fc8240fee90 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc823fff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc824195150 0x7fc82419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc823fff640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc824195150 0x7fc82419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42974/0 (socket says 172.21.15.118:42974) 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc823fff640 1 -- 172.21.15.118:0/1130361483 learned_addr learned my addr 172.21.15.118:0/1130361483 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc828cb7640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8240fee90 0x7fc824194c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc823fff640 1 -- 172.21.15.118:0/1130361483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8240fee90 msgr2=0x7fc824194c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc823fff640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8240fee90 0x7fc824194c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc823fff640 1 -- 172.21.15.118:0/1130361483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc810009560 con 0x7fc824195150 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc823fff640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc824195150 0x7fc82419a110 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc81002c960 tx=0x7fc81002c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:02.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc821ffb640 1 -- 172.21.15.118:0/1130361483 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc810004520 con 0x7fc824195150 2024-03-12T03:23:02.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82419a650 con 0x7fc824195150 2024-03-12T03:23:02.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc821ffb640 1 -- 172.21.15.118:0/1130361483 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc810003ab0 con 0x7fc824195150 2024-03-12T03:23:02.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.878+0000 7fc821ffb640 1 -- 172.21.15.118:0/1130361483 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8100055f0 con 0x7fc824195150 2024-03-12T03:23:02.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.879+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82419aba0 con 0x7fc824195150 2024-03-12T03:23:02.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.880+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82404e030 con 0x7fc824195150 2024-03-12T03:23:02.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.880+0000 7fc821ffb640 1 -- 172.21.15.118:0/1130361483 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fc810004e60 con 0x7fc824195150 2024-03-12T03:23:02.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.880+0000 7fc821ffb640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc81406c7f0 0x7fc81406ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:02.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.881+0000 7fc821ffb640 1 -- 172.21.15.118:0/1130361483 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fc8100adf50 con 0x7fc824195150 2024-03-12T03:23:02.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.881+0000 7fc828cb7640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc81406c7f0 0x7fc81406ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:02.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.881+0000 7fc828cb7640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc81406c7f0 0x7fc81406ecb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc818009bd0 tx=0x7fc818005f90 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:02.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:02.888+0000 7fc821ffb640 1 -- 172.21.15.118:0/1130361483 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fc8100e1960 con 0x7fc824195150 2024-03-12T03:23:03.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.076+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc82418e990 con 0x7fc824195150 2024-03-12T03:23:03.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.077+0000 7fc821ffb640 1 -- 172.21.15.118:0/1130361483 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+15192 (secure 0 0 0) 0x7fc810078470 con 0x7fc824195150 2024-03-12T03:23:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:23:03.080 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":47,"fsid":"9d8e7844-e01e-11ee-95c8-87774f69a715","created":"2024-03-12T03:15:23.858621+0000","modified":"2024-03-12T03:22:53.624359+0000","last_up_change":"2024-03-12T03:21:19.616038+0000","last_in_change":"2024-03-12T03:21:00.310107+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:20:04.838424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":6,"score_stable":6,"optimal_score":0.5,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-12T03:22:36.363854+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-12T03:22:36.658171+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96b74c65-8c14-4178-9e8d-b981032baa7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6803","nonce":1275003428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6805","nonce":1275003428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6809","nonce":1275003428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":1275003428},{"type":"v1","addr":"172.21.15.118:6807","nonce":1275003428}]},"public_addr":"172.21.15.118:6803/1275003428","cluster_addr":"172.21.15.118:6805/1275003428","heartbeat_back_addr":"172.21.15.118:6809/1275003428","heartbeat_front_addr":"172.21.15.118:6807/1275003428","state":["exists","up"]},{"osd":1,"uuid":"3c911034-856e-43e7-b267-2a6c63cf46fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6811","nonce":1159607125}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6813","nonce":1159607125}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6817","nonce":1159607125}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":1159607125},{"type":"v1","addr":"172.21.15.118:6815","nonce":1159607125}]},"public_addr":"172.21.15.118:6811/1159607125","cluster_addr":"172.21.15.118:6813/1159607125","heartbeat_back_addr":"172.21.15.118:6817/1159607125","heartbeat_front_addr":"172.21.15.118:6815/1159607125","state":["exists","up"]},{"osd":2,"uuid":"b4a69006-8aa7-40ea-8410-5645691cba22","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6819","nonce":3899760947}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6821","nonce":3899760947}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6825","nonce":3899760947}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":3899760947},{"type":"v1","addr":"172.21.15.118:6823","nonce":3899760947}]},"public_addr":"172.21.15.118:6819/3899760947","cluster_addr":"172.21.15.118:6821/3899760947","heartbeat_back_addr":"172.21.15.118:6825/3899760947","heartbeat_front_addr":"172.21.15.118:6823/3899760947","state":["exists","up"]},{"osd":3,"uuid":"3d46b2ef-fba5-4437-b107-9a321bb94874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6801","nonce":2319016844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6803","nonce":2319016844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6807","nonce":2319016844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":2319016844},{"type":"v1","addr":"172.21.15.177:6805","nonce":2319016844}]},"public_addr":"172.21.15.177:6801/2319016844","cluster_addr":"172.21.15.177:6803/2319016844","heartbeat_back_addr":"172.21.15.177:6807/2319016844","heartbeat_front_addr":"172.21.15.177:6805/2319016844","state":["exists","up"]},{"osd":4,"uuid":"b56220a9-5e7e-443a-b2c2-abcbca3b0c8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6809","nonce":411610750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6811","nonce":411610750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6815","nonce":411610750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":411610750},{"type":"v1","addr":"172.21.15.177:6813","nonce":411610750}]},"public_addr":"172.21.15.177:6809/411610750","cluster_addr":"172.21.15.177:6811/411610750","heartbeat_back_addr":"172.21.15.177:6815/411610750","heartbeat_front_addr":"172.21.15.177:6813/411610750","state":["exists","up"]},{"osd":5,"uuid":"0fe10ad0-409a-4af7-84da-9f6eee5214bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6817","nonce":3801270167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6819","nonce":3801270167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6823","nonce":3801270167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3801270167},{"type":"v1","addr":"172.21.15.177:6821","nonce":3801270167}]},"public_addr":"172.21.15.177:6817/3801270167","cluster_addr":"172.21.15.177:6819/3801270167","heartbeat_back_addr":"172.21.15.177:6823/3801270167","heartbeat_front_addr":"172.21.15.177:6821/3801270167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:04.091837+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:19:32.064473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:01.474842+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:25.475415+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:20:51.750854+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:21:17.974712+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6827/2160517509":"2024-03-13T03:22:53.624331+0000","172.21.15.118:6800/1066995478":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6826/2160517509":"2024-03-13T03:22:53.624331+0000","172.21.15.118:0/2477380828":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/1118679394":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/1647647336":"2024-03-13T03:17:48.514930+0000","172.21.15.118:6801/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/3517988861":"2024-03-13T03:17:48.514930+0000","172.21.15.118:0/3521891639":"2024-03-13T03:17:48.514930+0000","172.21.15.118:6800/1005219268":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/2012557212":"2024-03-13T03:15:57.609810+0000","172.21.15.118:0/287975169":"2024-03-13T03:15:57.609810+0000","172.21.15.118:6800/180481365":"2024-03-13T03:17:48.514930+0000","172.21.15.118:6801/1066995478":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/3667900250":"2024-03-13T03:16:31.649004+0000","172.21.15.118:0/1224337529":"2024-03-13T03:16:31.649004+0000","172.21.15.118:6801/180481365":"2024-03-13T03:17:48.514930+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc81406c7f0 msgr2=0x7fc81406ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc81406c7f0 0x7fc81406ecb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc818009bd0 tx=0x7fc818005f90 comp rx=0 tx=0).stop 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc824195150 msgr2=0x7fc82419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc824195150 0x7fc82419a110 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc81002c960 tx=0x7fc81002c750 comp rx=0 tx=0).stop 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 shutdown_connections 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fc81406c7f0 0x7fc81406ecb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc824195150 0x7fc82419a110 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 --2- 172.21.15.118:0/1130361483 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fc8240fee90 0x7fc824194c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 >> 172.21.15.118:0/1130361483 conn(0x7fc8240fa440 msgr2=0x7fc824103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:03.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 shutdown_connections 2024-03-12T03:23:03.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.079+0000 7fc82af1b640 1 -- 172.21.15.118:0/1130361483 wait complete. 2024-03-12T03:23:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:20:04.838424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 6, 'score_stable': 6, 'optimal_score': 0.5, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-12T03:22:36.363854+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-12T03:22:36.658171+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-12T03:23:03.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-12T03:23:03.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 -- 172.21.15.118:0/3133632798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 msgr2=0x7f699c1003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 --2- 172.21.15.118:0/3133632798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c1003c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6994009880 tx=0x7f699402bce0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 -- 172.21.15.118:0/3133632798 shutdown_connections 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 --2- 172.21.15.118:0/3133632798 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f699c101190 0x7f699c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 --2- 172.21.15.118:0/3133632798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c1003c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 -- 172.21.15.118:0/3133632798 >> 172.21.15.118:0/3133632798 conn(0x7f699c0fb530 msgr2=0x7f699c0fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 -- 172.21.15.118:0/3133632798 shutdown_connections 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 -- 172.21.15.118:0/3133632798 wait complete. 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 Processor -- start 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.199+0000 7f69a3451640 1 -- start start 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a3451640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a3451640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f699c101190 0x7f699c195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a3451640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c1965a0 con 0x7f699c0fffa0 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a3451640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c196710 con 0x7f699c101190 2024-03-12T03:23:03.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a11ed640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a11ed640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42978/0 (socket says 172.21.15.118:42978) 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a11ed640 1 -- 172.21.15.118:0/2042884421 learned_addr learned my addr 172.21.15.118:0/2042884421 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a09ec640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f699c101190 0x7f699c195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a11ed640 1 -- 172.21.15.118:0/2042884421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f699c101190 msgr2=0x7f699c195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a11ed640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f699c101190 0x7f699c195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a11ed640 1 -- 172.21.15.118:0/2042884421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6994009560 con 0x7f699c0fffa0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a11ed640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c195a90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6994000c00 tx=0x7f6994004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f699a7fc640 1 -- 172.21.15.118:0/2042884421 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69940035f0 con 0x7f699c0fffa0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c19b0f0 con 0x7f699c0fffa0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f699a7fc640 1 -- 172.21.15.118:0/2042884421 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6994003c10 con 0x7f699c0fffa0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f699a7fc640 1 -- 172.21.15.118:0/2042884421 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6994034440 con 0x7f699c0fffa0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.200+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c19b5f0 con 0x7f699c0fffa0 2024-03-12T03:23:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.201+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c18fb10 con 0x7f699c0fffa0 2024-03-12T03:23:03.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.202+0000 7f699a7fc640 1 -- 172.21.15.118:0/2042884421 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f6994004f60 con 0x7f699c0fffa0 2024-03-12T03:23:03.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.202+0000 7f699a7fc640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f698406c7f0 0x7f698406ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.202+0000 7f699a7fc640 1 -- 172.21.15.118:0/2042884421 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f69940ae0b0 con 0x7f699c0fffa0 2024-03-12T03:23:03.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.202+0000 7f69a09ec640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f698406c7f0 0x7f698406ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.203+0000 7f69a09ec640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f698406c7f0 0x7f698406ecb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f699c196f50 tx=0x7f699000b410 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:03.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.207+0000 7f699a7fc640 1 -- 172.21.15.118:0/2042884421 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f6994078710 con 0x7f699c0fffa0 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/475968729' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-12T03:23:03.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:03 smithi118 ceph-mon[29160]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:23:03.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.387+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f699c02c3f0 con 0x7f699c0fffa0 2024-03-12T03:23:03.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.388+0000 7f699a7fc640 1 -- 172.21.15.118:0/2042884421 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f699407bd30 con 0x7f699c0fffa0 2024-03-12T03:23:03.390 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f698406c7f0 msgr2=0x7f698406ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f698406c7f0 0x7f698406ecb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f699c196f50 tx=0x7f699000b410 comp rx=0 tx=0).stop 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 msgr2=0x7f699c195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c195a90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6994000c00 tx=0x7f6994004040 comp rx=0 tx=0).stop 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 shutdown_connections 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f699c101190 0x7f699c195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f698406c7f0 0x7f698406ecb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 --2- 172.21.15.118:0/2042884421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f699c0fffa0 0x7f699c195a90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 >> 172.21.15.118:0/2042884421 conn(0x7f699c0fb530 msgr2=0x7f699c104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:03.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 shutdown_connections 2024-03-12T03:23:03.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.390+0000 7f69a3451640 1 -- 172.21.15.118:0/2042884421 wait complete. 2024-03-12T03:23:03.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-12T03:23:03.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:03.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:03.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/475968729' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-12T03:23:03.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:03 smithi177 ceph-mon[37140]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 11 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.506+0000 7ff3083dd640 1 -- 172.21.15.118:0/1071751494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 msgr2=0x7ff3000ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 --2- 172.21.15.118:0/1071751494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff3000ff2b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009880 tx=0x7ff2f002bce0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 -- 172.21.15.118:0/1071751494 shutdown_connections 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 --2- 172.21.15.118:0/1071751494 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3000fffe0 0x7ff300100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 --2- 172.21.15.118:0/1071751494 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff3000ff2b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 -- 172.21.15.118:0/1071751494 >> 172.21.15.118:0/1071751494 conn(0x7ff3000fa440 msgr2=0x7ff3000fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 -- 172.21.15.118:0/1071751494 shutdown_connections 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 -- 172.21.15.118:0/1071751494 wait complete. 2024-03-12T03:23:03.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 Processor -- start 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 -- start start 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff300067cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3000fffe0 0x7ff30019a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30019ad40 con 0x7ff3000fee90 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff3083dd640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30019ae80 con 0x7ff3000fffe0 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff306179640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff300067cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff306179640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff300067cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42990/0 (socket says 172.21.15.118:42990) 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.507+0000 7ff306179640 1 -- 172.21.15.118:0/3061597820 learned_addr learned my addr 172.21.15.118:0/3061597820 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff305978640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3000fffe0 0x7ff30019a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff306179640 1 -- 172.21.15.118:0/3061597820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3000fffe0 msgr2=0x7ff30019a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff306179640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3000fffe0 0x7ff30019a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff306179640 1 -- 172.21.15.118:0/3061597820 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f0009560 con 0x7ff3000fee90 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff305978640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3000fffe0 0x7ff30019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T03:23:03.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff306179640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff300067cf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009430 tx=0x7ff2f0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:03.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff2f77fe640 1 -- 172.21.15.118:0/3061597820 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f00035f0 con 0x7ff3000fee90 2024-03-12T03:23:03.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff2f77fe640 1 -- 172.21.15.118:0/3061597820 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff2f0003c10 con 0x7ff3000fee90 2024-03-12T03:23:03.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30019b0d0 con 0x7ff3000fee90 2024-03-12T03:23:03.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff2f77fe640 1 -- 172.21.15.118:0/3061597820 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff2f0034440 con 0x7ff3000fee90 2024-03-12T03:23:03.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.508+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30019b650 con 0x7ff3000fee90 2024-03-12T03:23:03.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.509+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff30018e990 con 0x7ff3000fee90 2024-03-12T03:23:03.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.509+0000 7ff2f77fe640 1 -- 172.21.15.118:0/3061597820 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff2f0005060 con 0x7ff3000fee90 2024-03-12T03:23:03.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.510+0000 7ff2f77fe640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff2ec06c800 0x7ff2ec06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.510+0000 7ff305978640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff2ec06c800 0x7ff2ec06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.510+0000 7ff2f77fe640 1 -- 172.21.15.118:0/3061597820 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff2f00ae110 con 0x7ff3000fee90 2024-03-12T03:23:03.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.510+0000 7ff305978640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff2ec06c800 0x7ff2ec06ecc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff300067a60 tx=0x7ff2fc005e70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:03.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.515+0000 7ff2f77fe640 1 -- 172.21.15.118:0/3061597820 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff2f00786c0 con 0x7ff3000fee90 2024-03-12T03:23:03.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.708+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7ff300107a00 con 0x7ff3000fee90 2024-03-12T03:23:03.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.708+0000 7ff2f77fe640 1 -- 172.21.15.118:0/3061597820 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7ff2f007bce0 con 0x7ff3000fee90 2024-03-12T03:23:03.711 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff2ec06c800 msgr2=0x7ff2ec06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff2ec06c800 0x7ff2ec06ecc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff300067a60 tx=0x7ff2fc005e70 comp rx=0 tx=0).stop 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 msgr2=0x7ff300067cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff300067cf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff2f0009430 tx=0x7ff2f0004040 comp rx=0 tx=0).stop 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 shutdown_connections 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff3000fffe0 0x7ff30019a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff2ec06c800 0x7ff2ec06ecc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 --2- 172.21.15.118:0/3061597820 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3000fee90 0x7ff300067cf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 >> 172.21.15.118:0/3061597820 conn(0x7ff3000fa440 msgr2=0x7ff300103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:03.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 shutdown_connections 2024-03-12T03:23:03.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.711+0000 7ff3083dd640 1 -- 172.21.15.118:0/3061597820 wait complete. 2024-03-12T03:23:03.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-12T03:23:03.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 -- 172.21.15.118:0/4229747364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a00fee60 msgr2=0x7f00a00ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 --2- 172.21.15.118:0/4229747364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a00fee60 0x7f00a00ff280 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f009c009880 tx=0x7f009c02bce0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 -- 172.21.15.118:0/4229747364 shutdown_connections 2024-03-12T03:23:03.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 --2- 172.21.15.118:0/4229747364 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a0100040 0x7f00a01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 --2- 172.21.15.118:0/4229747364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a00fee60 0x7f00a00ff280 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 -- 172.21.15.118:0/4229747364 >> 172.21.15.118:0/4229747364 conn(0x7f00a00fa440 msgr2=0x7f00a00fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:03.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 -- 172.21.15.118:0/4229747364 shutdown_connections 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.836+0000 7f00a80b6640 1 -- 172.21.15.118:0/4229747364 wait complete. 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a80b6640 1 Processor -- start 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a80b6640 1 -- start start 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a80b6640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a00fee60 0x7f00a00689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a80b6640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a0100040 0x7f00a019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a80b6640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a019acf0 con 0x7f00a0100040 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a80b6640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a019ae60 con 0x7f00a00fee60 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5651640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a0100040 0x7f00a019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5651640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a0100040 0x7f00a019a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43008/0 (socket says 172.21.15.118:43008) 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5651640 1 -- 172.21.15.118:0/2596834583 learned_addr learned my addr 172.21.15.118:0/2596834583 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5e52640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a00fee60 0x7f00a00689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5651640 1 -- 172.21.15.118:0/2596834583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a00fee60 msgr2=0x7f00a00689e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:03.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5651640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a00fee60 0x7f00a00689e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:03.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5651640 1 -- 172.21.15.118:0/2596834583 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c009560 con 0x7f00a0100040 2024-03-12T03:23:03.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.837+0000 7f00a5651640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a0100040 0x7f00a019a7b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f009000e970 tx=0x7f009000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:03.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.838+0000 7f0096ffd640 1 -- 172.21.15.118:0/2596834583 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f009000cc90 con 0x7f00a0100040 2024-03-12T03:23:03.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.838+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a019b140 con 0x7f00a0100040 2024-03-12T03:23:03.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.838+0000 7f0096ffd640 1 -- 172.21.15.118:0/2596834583 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f009000cdf0 con 0x7f00a0100040 2024-03-12T03:23:03.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.838+0000 7f0096ffd640 1 -- 172.21.15.118:0/2596834583 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0090010430 con 0x7f00a0100040 2024-03-12T03:23:03.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.838+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a019b720 con 0x7f00a0100040 2024-03-12T03:23:03.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.839+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a018e9a0 con 0x7f00a0100040 2024-03-12T03:23:03.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.840+0000 7f0096ffd640 1 -- 172.21.15.118:0/2596834583 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f00900046c0 con 0x7f00a0100040 2024-03-12T03:23:03.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.840+0000 7f0096ffd640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f008c06c800 0x7f008c06ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:03.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.840+0000 7f0096ffd640 1 -- 172.21.15.118:0/2596834583 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f0090014070 con 0x7f00a0100040 2024-03-12T03:23:03.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.840+0000 7f00a5e52640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f008c06c800 0x7f008c06ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:03.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.841+0000 7f00a5e52640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f008c06c800 0x7f008c06ecc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f009c006010 tx=0x7f009c02c920 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:03.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:03.844+0000 7f0096ffd640 1 -- 172.21.15.118:0/2596834583 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f00900568c0 con 0x7f00a0100040 2024-03-12T03:23:04.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.023+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f00a0107a10 con 0x7f00a0100040 2024-03-12T03:23:04.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.024+0000 7f0096ffd640 1 -- 172.21.15.118:0/2596834583 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0090059ee0 con 0x7f00a0100040 2024-03-12T03:23:04.026 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-12T03:23:04.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f008c06c800 msgr2=0x7f008c06ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:04.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f008c06c800 0x7f008c06ecc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f009c006010 tx=0x7f009c02c920 comp rx=0 tx=0).stop 2024-03-12T03:23:04.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a0100040 msgr2=0x7f00a019a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:04.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a0100040 0x7f00a019a7b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f009000e970 tx=0x7f009000ed70 comp rx=0 tx=0).stop 2024-03-12T03:23:04.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 shutdown_connections 2024-03-12T03:23:04.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f008c06c800 0x7f008c06ecc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00a0100040 0x7f00a019a7b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 --2- 172.21.15.118:0/2596834583 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f00a00fee60 0x7f00a00689e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 >> 172.21.15.118:0/2596834583 conn(0x7f00a00fa440 msgr2=0x7f00a01036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:04.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 shutdown_connections 2024-03-12T03:23:04.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.026+0000 7f00a80b6640 1 -- 172.21.15.118:0/2596834583 wait complete. 2024-03-12T03:23:04.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-12T03:23:04.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 -- 172.21.15.118:0/7529331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fee80 msgr2=0x7f99140ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:04.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 --2- 172.21.15.118:0/7529331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fee80 0x7f99140ff2a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9910009880 tx=0x7f991002bce0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 -- 172.21.15.118:0/7529331 shutdown_connections 2024-03-12T03:23:04.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 --2- 172.21.15.118:0/7529331 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99140fffd0 0x7f9914100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 --2- 172.21.15.118:0/7529331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fee80 0x7f99140ff2a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 -- 172.21.15.118:0/7529331 >> 172.21.15.118:0/7529331 conn(0x7f99140fa410 msgr2=0x7f99140fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:04.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 -- 172.21.15.118:0/7529331 shutdown_connections 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.141+0000 7f991c4e0640 1 -- 172.21.15.118:0/7529331 wait complete. 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991c4e0640 1 Processor -- start 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991c4e0640 1 -- start start 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991c4e0640 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99140fee80 0x7f9914068e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991c4e0640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fffd0 0x7f991419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991c4e0640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991419ad30 con 0x7f99140fffd0 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991c4e0640 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991419ae70 con 0x7f99140fee80 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f9919a7b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fffd0 0x7f991419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f9919a7b640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fffd0 0x7f991419a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43018/0 (socket says 172.21.15.118:43018) 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f9919a7b640 1 -- 172.21.15.118:0/3888872416 learned_addr learned my addr 172.21.15.118:0/3888872416 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991a27c640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99140fee80 0x7f9914068e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f9919a7b640 1 -- 172.21.15.118:0/3888872416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99140fee80 msgr2=0x7f9914068e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f9919a7b640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99140fee80 0x7f9914068e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f9919a7b640 1 -- 172.21.15.118:0/3888872416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9910009560 con 0x7f99140fffd0 2024-03-12T03:23:04.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f9919a7b640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fffd0 0x7f991419a7f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f990400d720 tx=0x7f990400dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:04.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f990b7fe640 1 -- 172.21.15.118:0/3888872416 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99040041f0 con 0x7f99140fffd0 2024-03-12T03:23:04.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f990b7fe640 1 -- 172.21.15.118:0/3888872416 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9904004350 con 0x7f99140fffd0 2024-03-12T03:23:04.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f990b7fe640 1 -- 172.21.15.118:0/3888872416 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9904003df0 con 0x7f99140fffd0 2024-03-12T03:23:04.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.142+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f991419b120 con 0x7f99140fffd0 2024-03-12T03:23:04.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.143+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f991419b700 con 0x7f99140fffd0 2024-03-12T03:23:04.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.143+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f991418e960 con 0x7f99140fffd0 2024-03-12T03:23:04.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.144+0000 7f990b7fe640 1 -- 172.21.15.118:0/3888872416 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f99040035f0 con 0x7f99140fffd0 2024-03-12T03:23:04.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.144+0000 7f990b7fe640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f990006c7f0 0x7f990006ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:04.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.144+0000 7f990b7fe640 1 -- 172.21.15.118:0/3888872416 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f990408b5f0 con 0x7f99140fffd0 2024-03-12T03:23:04.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.144+0000 7f991a27c640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f990006c7f0 0x7f990006ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:04.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.145+0000 7f991a27c640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f990006c7f0 0x7f990006ecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9910009230 tx=0x7f9910005fd0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:04.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.149+0000 7f990b7fe640 1 -- 172.21.15.118:0/3888872416 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9904055cd0 con 0x7f99140fffd0 2024-03-12T03:23:04.290 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:04 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1130361483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:23:04.290 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:04 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2042884421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:23:04.290 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:04.290 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:04.290 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:04 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3061597820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-12T03:23:04.290 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:04 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2596834583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-12T03:23:04.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.365+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9914107c00 con 0x7f99140fffd0 2024-03-12T03:23:04.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.366+0000 7f990b7fe640 1 -- 172.21.15.118:0/3888872416 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4865 (secure 0 0 0) 0x7f99040592f0 con 0x7f99140fffd0 2024-03-12T03:23:04.368 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-12T03:23:04.368 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":18,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-12T03:22:37.425947+0000","modified":"2024-03-12T03:22:57.649892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14486,"mds_1":24279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14486":{"gid":14486,"name":"cephfs.smithi118.oysrod","rank":0,"incarnation":13,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6829/2437365950","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2437365950},{"type":"v1","addr":"172.21.15.118:6829","nonce":2437365950}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14498":{"gid":14498,"name":"cephfs.smithi118.wouiai","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6827/931262818","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":931262818},{"type":"v1","addr":"172.21.15.118:6827","nonce":931262818}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24279":{"gid":24279,"name":"cephfs.smithi177.kbzhid","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.177:6825/4286002121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":4286002121},{"type":"v1","addr":"172.21.15.177:6825","nonce":4286002121}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24291":{"gid":24291,"name":"cephfs.smithi177.azybll","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.177:6827/1551126945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":1551126945},{"type":"v1","addr":"172.21.15.177:6827","nonce":1551126945}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-12T03:23:04.369 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 18 2024-03-12T03:23:04.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f990006c7f0 msgr2=0x7f990006ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f990006c7f0 0x7f990006ecb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9910009230 tx=0x7f9910005fd0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fffd0 msgr2=0x7f991419a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fffd0 0x7f991419a7f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f990400d720 tx=0x7f990400dbe0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 shutdown_connections 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f990006c7f0 0x7f990006ecb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f99140fffd0 0x7f991419a7f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 --2- 172.21.15.118:0/3888872416 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f99140fee80 0x7f9914068e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 >> 172.21.15.118:0/3888872416 conn(0x7f99140fa410 msgr2=0x7f99141036b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 shutdown_connections 2024-03-12T03:23:04.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:04.368+0000 7f991c4e0640 1 -- 172.21.15.118:0/3888872416 wait complete. 2024-03-12T03:23:04.381 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 18, 'max_mds': 2, 'flags': 50} 2024-03-12T03:23:04.381 INFO:teuthology.run_tasks:Running task kclient... 2024-03-12T03:23:04.411 INFO:tasks.kclient:Mounting kernel clients... 2024-03-12T03:23:04.411 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-12T03:23:04.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:04.411 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-03-12T03:23:04.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:04.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-03-12T03:23:04.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:04 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1130361483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:23:04.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:04 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2042884421' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:23:04.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:04.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:04.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:04 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3061597820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-12T03:23:04.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:04 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2596834583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-12T03:23:04.498 INFO:teuthology.orchestra.run.smithi118.stderr:Cannot find device "ceph-brx" 2024-03-12T03:23:04.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:23:04.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:04.501 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-03-12T03:23:04.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:04.517 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link delete ceph-brx 2024-03-12T03:23:04.588 INFO:teuthology.orchestra.run.smithi177.stderr:Cannot find device "ceph-brx" 2024-03-12T03:23:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:23:04.589 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-12T03:23:04.589 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-12T03:23:04.589 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:04.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-12T03:23:05.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:05 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:05.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:05 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:05.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:05 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3888872416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T03:23:05.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:05 smithi118 ceph-mon[29160]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 9.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:23:05.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:05 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:05.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:05 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:05.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:05 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3888872416' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-12T03:23:05.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:05 smithi177 ceph-mon[37140]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 9.5 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:23:05.964 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:07.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:07 smithi118 ceph-mon[29160]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:23:07.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:07 smithi177 ceph-mon[37140]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.6 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 -- 172.21.15.118:0/2241987129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c102660 msgr2=0x7f968c10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 --2- 172.21.15.118:0/2241987129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c102660 0x7f968c10ab50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f967c009910 tx=0x7f967c009c20 comp rx=0 tx=0).stop 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 -- 172.21.15.118:0/2241987129 shutdown_connections 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 --2- 172.21.15.118:0/2241987129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c102660 0x7f968c10ab50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 --2- 172.21.15.118:0/2241987129 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c101d40 0x7f968c102120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 -- 172.21.15.118:0/2241987129 >> 172.21.15.118:0/2241987129 conn(0x7f968c0fb4d0 msgr2=0x7f968c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 -- 172.21.15.118:0/2241987129 shutdown_connections 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.293+0000 7f9693a12700 1 -- 172.21.15.118:0/2241987129 wait complete. 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9693a12700 1 Processor -- start 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9693a12700 1 -- start start 2024-03-12T03:23:08.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9693a12700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c101d40 0x7f968c19c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9693a12700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c102660 0x7f968c19cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9693a12700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c19d350 con 0x7f968c101d40 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9693a12700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c196930 con 0x7f968c102660 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96917ae700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c101d40 0x7f968c19c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96917ae700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c101d40 0x7f968c19c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45338/0 (socket says 172.21.15.118:45338) 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96917ae700 1 -- 172.21.15.118:0/3647171473 learned_addr learned my addr 172.21.15.118:0/3647171473 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9690fad700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c102660 0x7f968c19cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96917ae700 1 -- 172.21.15.118:0/3647171473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c102660 msgr2=0x7f968c19cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96917ae700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c102660 0x7f968c19cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96917ae700 1 -- 172.21.15.118:0/3647171473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c0095f0 con 0x7f968c101d40 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96917ae700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c101d40 0x7f968c19c7f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f968400ba50 tx=0x7f968400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:08.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96827fc700 1 -- 172.21.15.118:0/3647171473 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f968400ca00 con 0x7f968c101d40 2024-03-12T03:23:08.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f96827fc700 1 -- 172.21.15.118:0/3647171473 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f968400cb60 con 0x7f968c101d40 2024-03-12T03:23:08.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.294+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c196c10 con 0x7f968c101d40 2024-03-12T03:23:08.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.295+0000 7f96827fc700 1 -- 172.21.15.118:0/3647171473 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96840075a0 con 0x7f968c101d40 2024-03-12T03:23:08.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.295+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c197160 con 0x7f968c101d40 2024-03-12T03:23:08.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.295+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968c105460 con 0x7f968c101d40 2024-03-12T03:23:08.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.296+0000 7f96827fc700 1 -- 172.21.15.118:0/3647171473 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f968400ccd0 con 0x7f968c101d40 2024-03-12T03:23:08.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.297+0000 7f96827fc700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f967806c880 0x7f967806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:08.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.297+0000 7f96827fc700 1 -- 172.21.15.118:0/3647171473 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f9684006e40 con 0x7f968c101d40 2024-03-12T03:23:08.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.297+0000 7f9690fad700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f967806c880 0x7f967806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:08.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.297+0000 7f9690fad700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f967806c880 0x7f967806ed40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f968c197af0 tx=0x7f967c01a040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:08.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.301+0000 7f96827fc700 1 -- 172.21.15.118:0/3647171473 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f9684059590 con 0x7f968c101d40 2024-03-12T03:23:08.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.537+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f968c197df0 con 0x7f968c101d40 2024-03-12T03:23:08.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.538+0000 7f96827fc700 1 -- 172.21.15.118:0/3647171473 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f9684059120 con 0x7f968c101d40 2024-03-12T03:23:08.540 INFO:teuthology.orchestra.run.smithi118.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-12T03:23:08.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f967806c880 msgr2=0x7f967806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:08.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f967806c880 0x7f967806ed40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f968c197af0 tx=0x7f967c01a040 comp rx=0 tx=0).stop 2024-03-12T03:23:08.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c101d40 msgr2=0x7f968c19c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:08.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c101d40 0x7f968c19c7f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f968400ba50 tx=0x7f968400be10 comp rx=0 tx=0).stop 2024-03-12T03:23:08.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 shutdown_connections 2024-03-12T03:23:08.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f967806c880 0x7f967806ed40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:08.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f968c101d40 0x7f968c19c7f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:08.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 --2- 172.21.15.118:0/3647171473 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f968c102660 0x7f968c19cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:08.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 >> 172.21.15.118:0/3647171473 conn(0x7f968c0fb4d0 msgr2=0x7f968c103030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:08.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 shutdown_connections 2024-03-12T03:23:08.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:08.540+0000 7f9693a12700 1 -- 172.21.15.118:0/3647171473 wait complete. 2024-03-12T03:23:09.173 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:09.173 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:09.174 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:09.174 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:09.174 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:09 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:09.174 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:09 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/3647171473' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T03:23:09.174 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:09 smithi118 ceph-mon[29160]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:23:09.195 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-12T03:23:09.195 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-12T03:23:09.195 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi118.front.sepia.ceph.com 2024-03-12T03:23:09.195 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-12T03:23:09.195 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:09.195 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-12T03:23:09.195 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-12T03:23:09.196 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-12T03:23:09.196 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-12T03:23:09.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.196 DEBUG:teuthology.orchestra.run.smithi118:> ip addr 2024-03-12T03:23:09.212 INFO:teuthology.orchestra.run.smithi118.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-12T03:23:09.212 INFO:teuthology.orchestra.run.smithi118.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-12T03:23:09.212 INFO:teuthology.orchestra.run.smithi118.stdout: inet 127.0.0.1/8 scope host lo 2024-03-12T03:23:09.212 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 ::1/128 scope host 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cc brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f0 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cd brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f1 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:24 brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:25 brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: inet 172.21.15.118/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-12T03:23:09.213 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft 42023sec preferred_lft 42023sec 2024-03-12T03:23:09.214 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::ec4:7aff:fe8f:d725/64 scope link 2024-03-12T03:23:09.214 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-03-12T03:23:09.214 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-12T03:23:09.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.214 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-12T03:23:09.214 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-12T03:23:09.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link add name ceph-brx type bridge 2024-03-12T03:23:09.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip addr flush dev ceph-brx 2024-03-12T03:23:09.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set ceph-brx up 2024-03-12T03:23:09.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-12T03:23:09.215 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-12T03:23:09.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.376 DEBUG:teuthology.orchestra.run.smithi118:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-12T03:23:09.402 INFO:teuthology.orchestra.run.smithi118.stdout:1 2024-03-12T03:23:09.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.403 DEBUG:teuthology.orchestra.run.smithi118:> route 2024-03-12T03:23:09.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:09.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:09.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:09.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:09.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:09 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:09.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:09 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/3647171473' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T03:23:09.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:09 smithi177 ceph-mon[37140]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 8.4 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:23:09.471 INFO:teuthology.orchestra.run.smithi118.stdout:Kernel IP routing table 2024-03-12T03:23:09.471 INFO:teuthology.orchestra.run.smithi118.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-12T03:23:09.471 INFO:teuthology.orchestra.run.smithi118.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-12T03:23:09.471 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-12T03:23:09.471 INFO:teuthology.orchestra.run.smithi118.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-12T03:23:09.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.472 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-12T03:23:09.472 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-12T03:23:09.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-12T03:23:09.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-12T03:23:09.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-12T03:23:09.472 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-12T03:23:09.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.787 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-03-12T03:23:09.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.803 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list-id 2024-03-12T03:23:09.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.858 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-12T03:23:09.858 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-12T03:23:09.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-12T03:23:09.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-12T03:23:09.858 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-12T03:23:09.939 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-12T03:23:09.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-12T03:23:09.940 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-12T03:23:10.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:10.074 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-12T03:23:10.074 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-12T03:23:10.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set brx.0 up 2024-03-12T03:23:10.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set dev brx.0 master ceph-brx 2024-03-12T03:23:10.074 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-12T03:23:10.115 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T03:23:10.115 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:10.171 INFO:teuthology.orchestra.run.smithi118.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-12T03:23:10.171 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T03:23:10.171 DEBUG:teuthology.orchestra.run.smithi118:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:10.225 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-12T03:23:10.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:10.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-12T03:23:10.327 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-12T03:23:10.328 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-12T03:23:10.328 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-03-12T03:23:10.434 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-12T03:23:10.434 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-12T03:23:10.434 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-03-12T03:23:10.434 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.118:3300,172.21.15.177:3300" 2024-03-12T03:23:10.434 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-12T03:23:10.439 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-12T03:23:10.439 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-12T03:23:10.439 INFO:teuthology.orchestra.run.smithi118.stdout:invalid new device string format 2024-03-12T03:23:10.439 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.118:3300,172.21.15.177:3300" 2024-03-12T03:23:10.439 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-12T03:23:10.439 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.118:6789,172.21.15.177:6789" 2024-03-12T03:23:10.439 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.118:6789,172.21.15.177:6789:/ 2024-03-12T03:23:10.440 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=9d8e7844-e01e-11ee-95c8-87774f69a715" will pass to kernel 2024-03-12T03:23:10.440 INFO:teuthology.orchestra.run.smithi118.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-12T03:23:10.440 INFO:teuthology.orchestra.run.smithi118.stdout: You just mounted a file system that supports labels which does not 2024-03-12T03:23:10.440 INFO:teuthology.orchestra.run.smithi118.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-12T03:23:10.440 INFO:teuthology.orchestra.run.smithi118.stdout: applications will generate AVC messages and not be allowed access to 2024-03-12T03:23:10.440 INFO:teuthology.orchestra.run.smithi118.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-12T03:23:10.443 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-12T03:23:10.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:10.443 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:10.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:10.475 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T03:23:10.535 INFO:teuthology.orchestra.run.smithi118.stdout:client14536 2024-03-12T03:23:10.536 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:23:10.536 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T03:23:10.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:10.592 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T03:23:10.652 INFO:teuthology.orchestra.run.smithi118.stdout:client14536 2024-03-12T03:23:10.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:10.653 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T03:23:10.712 INFO:teuthology.orchestra.run.smithi118.stdout:client14536 2024-03-12T03:23:10.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:10.712 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715.client14536/status') 2024-03-12T03:23:10.781 INFO:teuthology.orchestra.run.smithi118.stdout:instance: client.14536 (3)192.168.0.1:0/2358854247 2024-03-12T03:23:10.782 INFO:teuthology.orchestra.run.smithi118.stdout:blocklisted: false 2024-03-12T03:23:10.782 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records in 2024-03-12T03:23:10.782 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records out 2024-03-12T03:23:10.782 INFO:teuthology.orchestra.run.smithi118.stderr:70 bytes copied, 2.3315e-05 s, 3.0 MB/s 2024-03-12T03:23:10.783 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-12T03:23:10.783 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-12T03:23:10.784 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:10.784 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-12T03:23:11.030 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:11.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:11 smithi118 ceph-mon[29160]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T03:23:11.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:11 smithi177 ceph-mon[37140]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-12T03:23:12.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 -- 172.21.15.118:0/3309327663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073a70 msgr2=0x7f96d0111530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 --2- 172.21.15.118:0/3309327663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073a70 0x7f96d0111530 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009910 tx=0x7f96c0009c20 comp rx=0 tx=0).stop 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 -- 172.21.15.118:0/3309327663 shutdown_connections 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 --2- 172.21.15.118:0/3309327663 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073a70 0x7f96d0111530 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 --2- 172.21.15.118:0/3309327663 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d0073150 0x7f96d0073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 -- 172.21.15.118:0/3309327663 >> 172.21.15.118:0/3309327663 conn(0x7f96d00fc590 msgr2=0x7f96d00fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 -- 172.21.15.118:0/3309327663 shutdown_connections 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.685+0000 7f96d7796700 1 -- 172.21.15.118:0/3309327663 wait complete. 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d7796700 1 Processor -- start 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d7796700 1 -- start start 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d7796700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073150 0x7f96d019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d7796700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d0073a70 0x7f96d019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:12.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d7796700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d019dbd0 con 0x7f96d0073150 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d7796700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d01a18a0 con 0x7f96d0073a70 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d5532700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073150 0x7f96d019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d5532700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073150 0x7f96d019cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45352/0 (socket says 172.21.15.118:45352) 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d5532700 1 -- 172.21.15.118:0/1681985161 learned_addr learned my addr 172.21.15.118:0/1681985161 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d5532700 1 -- 172.21.15.118:0/1681985161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d0073a70 msgr2=0x7f96d019d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d5532700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d0073a70 0x7f96d019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d5532700 1 -- 172.21.15.118:0/1681985161 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00095f0 con 0x7f96d0073150 2024-03-12T03:23:12.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.686+0000 7f96d5532700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073150 0x7f96d019cfb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00d8d0 tx=0x7f96cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:12.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.687+0000 7f96c67fc700 1 -- 172.21.15.118:0/1681985161 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96cc009940 con 0x7f96d0073150 2024-03-12T03:23:12.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.687+0000 7f96c67fc700 1 -- 172.21.15.118:0/1681985161 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96cc010460 con 0x7f96d0073150 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.687+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d01a1b80 con 0x7f96d0073150 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.687+0000 7f96c67fc700 1 -- 172.21.15.118:0/1681985161 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96cc00f5f0 con 0x7f96d0073150 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.687+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d01a20d0 con 0x7f96d0073150 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.688+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d0074890 con 0x7f96d0073150 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.688+0000 7f96c67fc700 1 -- 172.21.15.118:0/1681985161 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f96cc009aa0 con 0x7f96d0073150 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.688+0000 7f96c67fc700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f96bc06c880 0x7f96bc06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.689+0000 7f96c67fc700 1 -- 172.21.15.118:0/1681985161 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f96cc08bbf0 con 0x7f96d0073150 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.689+0000 7f96d4d31700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f96bc06c880 0x7f96bc06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:12.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.689+0000 7f96d4d31700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f96bc06c880 0x7f96bc06ed40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f96c000b5c0 tx=0x7f96c0005e20 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:12.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.693+0000 7f96c67fc700 1 -- 172.21.15.118:0/1681985161 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f96cc056220 con 0x7f96d0073150 2024-03-12T03:23:12.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.930+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f96d01a2400 con 0x7f96d0073150 2024-03-12T03:23:12.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.930+0000 7f96c67fc700 1 -- 172.21.15.118:0/1681985161 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+83 (secure 0 0 0) 0x7f96cc059840 con 0x7f96d0073150 2024-03-12T03:23:12.933 INFO:teuthology.orchestra.run.smithi118.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f96bc06c880 msgr2=0x7f96bc06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f96bc06c880 0x7f96bc06ed40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f96c000b5c0 tx=0x7f96c0005e20 comp rx=0 tx=0).stop 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073150 msgr2=0x7f96d019cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073150 0x7f96d019cfb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00d8d0 tx=0x7f96cc00dc90 comp rx=0 tx=0).stop 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 shutdown_connections 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f96bc06c880 0x7f96bc06ed40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d0073150 0x7f96d019cfb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 --2- 172.21.15.118:0/1681985161 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f96d0073a70 0x7f96d019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 >> 172.21.15.118:0/1681985161 conn(0x7f96d00fc590 msgr2=0x7f96d01007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 shutdown_connections 2024-03-12T03:23:12.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:12.932+0000 7f96d7796700 1 -- 172.21.15.118:0/1681985161 wait complete. 2024-03-12T03:23:13.586 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-12T03:23:13.586 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-12T03:23:13.586 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi177.front.sepia.ceph.com 2024-03-12T03:23:13.587 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-12T03:23:13.587 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:13.587 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-12T03:23:13.587 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-12T03:23:13.587 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-12T03:23:13.587 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-12T03:23:13.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:13.587 DEBUG:teuthology.orchestra.run.smithi177:> ip addr 2024-03-12T03:23:13.606 INFO:teuthology.orchestra.run.smithi177.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-12T03:23:13.606 INFO:teuthology.orchestra.run.smithi177.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-12T03:23:13.606 INFO:teuthology.orchestra.run.smithi177.stdout: inet 127.0.0.1/8 scope host lo 2024-03-12T03:23:13.606 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-12T03:23:13.606 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 ::1/128 scope host 2024-03-12T03:23:13.606 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:40 brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f0 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:d9:19:41 brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: altname enp7s0f1 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4a brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: link/ether 0c:c4:7a:ea:b0:4b brd ff:ff:ff:ff:ff:ff 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: inet 172.21.15.177/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft 42029sec preferred_lft 42029sec 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: inet6 fe80::ec4:7aff:feea:b04b/64 scope link 2024-03-12T03:23:13.607 INFO:teuthology.orchestra.run.smithi177.stdout: valid_lft forever preferred_lft forever 2024-03-12T03:23:13.608 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-12T03:23:13.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:13.608 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-12T03:23:13.608 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-12T03:23:13.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add name ceph-brx type bridge 2024-03-12T03:23:13.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip addr flush dev ceph-brx 2024-03-12T03:23:13.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set ceph-brx up 2024-03-12T03:23:13.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-12T03:23:13.608 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-12T03:23:13.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:13.776 DEBUG:teuthology.orchestra.run.smithi177:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-12T03:23:13.806 INFO:teuthology.orchestra.run.smithi177.stdout:1 2024-03-12T03:23:13.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:13.808 DEBUG:teuthology.orchestra.run.smithi177:> route 2024-03-12T03:23:13.873 INFO:teuthology.orchestra.run.smithi177.stdout:Kernel IP routing table 2024-03-12T03:23:13.890 INFO:teuthology.orchestra.run.smithi177.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-12T03:23:13.890 INFO:teuthology.orchestra.run.smithi177.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-12T03:23:13.890 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-12T03:23:13.890 INFO:teuthology.orchestra.run.smithi177.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-12T03:23:13.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:13.890 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-12T03:23:13.890 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-12T03:23:13.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-12T03:23:13.890 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-12T03:23:13.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-12T03:23:13.891 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-12T03:23:13.938 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:13 smithi177 ceph-mon[37140]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:23:13.938 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:13 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1681985161' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T03:23:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:13 smithi118 ceph-mon[29160]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 3.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:23:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:13 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1681985161' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-12T03:23:14.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.148 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list 2024-03-12T03:23:14.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.206 DEBUG:teuthology.orchestra.run.smithi177:> ip netns list-id 2024-03-12T03:23:14.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.262 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-12T03:23:14.262 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-12T03:23:14.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-12T03:23:14.262 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-12T03:23:14.262 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-12T03:23:14.346 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-12T03:23:14.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-12T03:23:14.347 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-12T03:23:14.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.443 DEBUG:teuthology.orchestra.run.smithi177:> (cd / && exec bash -c ' 2024-03-12T03:23:14.443 DEBUG:teuthology.orchestra.run.smithi177:> set -e 2024-03-12T03:23:14.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set brx.0 up 2024-03-12T03:23:14.443 DEBUG:teuthology.orchestra.run.smithi177:> sudo ip link set dev brx.0 master ceph-brx 2024-03-12T03:23:14.444 DEBUG:teuthology.orchestra.run.smithi177:> ') 2024-03-12T03:23:14.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T03:23:14.485 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:14.540 INFO:teuthology.orchestra.run.smithi177.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-12T03:23:14.540 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-12T03:23:14.540 DEBUG:teuthology.orchestra.run.smithi177:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:14.594 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-12T03:23:14.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-12T03:23:14.687 INFO:teuthology.orchestra.run.smithi177.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-12T03:23:14.687 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-12T03:23:14.687 INFO:teuthology.orchestra.run.smithi177.stdout:invalid new device string format 2024-03-12T03:23:14.789 INFO:teuthology.orchestra.run.smithi177.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-12T03:23:14.790 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-12T03:23:14.790 INFO:teuthology.orchestra.run.smithi177.stdout:invalid new device string format 2024-03-12T03:23:14.790 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: resolved to: "172.21.15.118:3300,172.21.15.177:3300" 2024-03-12T03:23:14.790 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-12T03:23:14.794 INFO:teuthology.orchestra.run.smithi177.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-12T03:23:14.794 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-12T03:23:14.794 INFO:teuthology.orchestra.run.smithi177.stdout:invalid new device string format 2024-03-12T03:23:14.794 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: resolved to: "172.21.15.118:3300,172.21.15.177:3300" 2024-03-12T03:23:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-12T03:23:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: resolved to: "172.21.15.118:6789,172.21.15.177:6789" 2024-03-12T03:23:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.118:6789,172.21.15.177:6789:/ 2024-03-12T03:23:14.795 INFO:teuthology.orchestra.run.smithi177.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=9d8e7844-e01e-11ee-95c8-87774f69a715" will pass to kernel 2024-03-12T03:23:14.796 INFO:teuthology.orchestra.run.smithi177.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-12T03:23:14.796 INFO:teuthology.orchestra.run.smithi177.stdout: You just mounted a file system that supports labels which does not 2024-03-12T03:23:14.796 INFO:teuthology.orchestra.run.smithi177.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-12T03:23:14.796 INFO:teuthology.orchestra.run.smithi177.stdout: applications will generate AVC messages and not be allowed access to 2024-03-12T03:23:14.796 INFO:teuthology.orchestra.run.smithi177.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-12T03:23:14.797 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-12T03:23:14.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.798 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:14.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.825 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T03:23:14.882 INFO:teuthology.orchestra.run.smithi177.stdout:client24325 2024-03-12T03:23:14.882 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:23:14.882 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T03:23:14.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.939 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T03:23:14.995 INFO:teuthology.orchestra.run.smithi177.stdout:client24325 2024-03-12T03:23:14.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:14.996 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-12T03:23:15.052 INFO:teuthology.orchestra.run.smithi177.stdout:client24325 2024-03-12T03:23:15.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:23:15.052 DEBUG:teuthology.orchestra.run.smithi177:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715.client24325/status') 2024-03-12T03:23:15.117 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:14 smithi177 ceph-mon[37140]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-12T03:23:15.119 INFO:teuthology.orchestra.run.smithi177.stdout:instance: client.24325 (3)192.168.0.1:0/2201058973 2024-03-12T03:23:15.120 INFO:teuthology.orchestra.run.smithi177.stdout:blocklisted: false 2024-03-12T03:23:15.120 INFO:teuthology.orchestra.run.smithi177.stderr:0+1 records in 2024-03-12T03:23:15.120 INFO:teuthology.orchestra.run.smithi177.stderr:0+1 records out 2024-03-12T03:23:15.120 INFO:teuthology.orchestra.run.smithi177.stderr:70 bytes copied, 3.7112e-05 s, 1.9 MB/s 2024-03-12T03:23:15.121 INFO:teuthology.run_tasks:Running task print... 2024-03-12T03:23:15.131 INFO:teuthology.task.print:**** done client 2024-03-12T03:23:15.131 INFO:teuthology.run_tasks:Running task parallel... 2024-03-12T03:23:15.150 INFO:teuthology.task.parallel:starting parallel... 2024-03-12T03:23:15.151 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-12T03:23:15.151 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-12T03:23:15.152 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:23:15.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-12T03:23:15.152 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-12T03:23:15.153 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-12T03:23:15.156 INFO:tasks.workunit:Pulling workunits from ref 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-12T03:23:15.156 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-12T03:23:15.156 INFO:tasks.workunit:timeout=3h 2024-03-12T03:23:15.156 INFO:tasks.workunit:cleanup=True 2024-03-12T03:23:15.156 DEBUG:teuthology.orchestra.run.smithi118:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:15.173 INFO:teuthology.orchestra.run.smithi118.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:15.173 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-12T03:23:15.174 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5ch/92d Inode: 1 Links: 2 2024-03-12T03:23:15.174 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:23:15.174 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-12T03:23:15.174 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-12T03:23:15.174 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-12 03:22:42.672585184 +0000 2024-03-12T03:23:15.174 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-12 03:23:10.467697436 +0000 2024-03-12T03:23:15.174 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-12 03:22:42.672585184 +0000 2024-03-12T03:23:15.174 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-12T03:23:15.174 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-12T03:23:15.183 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:14 smithi118 ceph-mon[29160]: pgmap v198: 65 pgs: 65 active+clean; 581 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-12T03:23:15.214 DEBUG:teuthology.orchestra.run.smithi177:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:15.231 INFO:teuthology.orchestra.run.smithi177.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:15.231 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-12T03:23:15.231 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 7eh/126d Inode: 1 Links: 3 2024-03-12T03:23:15.231 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:23:15.231 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-12T03:23:15.232 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-12T03:23:15.232 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-12 03:23:15.206597767 +0000 2024-03-12T03:23:15.232 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-12 03:23:15.206597767 +0000 2024-03-12T03:23:15.232 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-12 03:22:42.672585184 +0000 2024-03-12T03:23:15.232 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-12T03:23:15.232 DEBUG:teuthology.orchestra.run.smithi177:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-12T03:23:15.305 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-12T03:23:15.306 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 1c349451176cc5b4ebfb24b22eaaa754e05cff6c 2024-03-12T03:23:15.322 INFO:tasks.workunit.client.0.smithi118.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-12T03:23:15.361 INFO:tasks.workunit.client.1.smithi177.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-12T03:23:15.383 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:16.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 -- 172.21.15.118:0/4002927162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c100ab0 msgr2=0x7f910c100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:16.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 --2- 172.21.15.118:0/4002927162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c100ab0 0x7f910c100ed0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f90fc009910 tx=0x7f90fc009c20 comp rx=0 tx=0).stop 2024-03-12T03:23:16.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 -- 172.21.15.118:0/4002927162 shutdown_connections 2024-03-12T03:23:16.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 --2- 172.21.15.118:0/4002927162 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f910c101cb0 0x7f910c102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 --2- 172.21.15.118:0/4002927162 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c100ab0 0x7f910c100ed0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 -- 172.21.15.118:0/4002927162 >> 172.21.15.118:0/4002927162 conn(0x7f910c0fc090 msgr2=0x7f910c0fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 -- 172.21.15.118:0/4002927162 shutdown_connections 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 -- 172.21.15.118:0/4002927162 wait complete. 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 Processor -- start 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.930+0000 7f9110d2c700 1 -- start start 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9110d2c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f910c100ab0 0x7f910c1964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9110d2c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c101cb0 0x7f910c1969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9110d2c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c197000 con 0x7f910c101cb0 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9110d2c700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c197140 con 0x7f910c100ab0 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9109d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c101cb0 0x7f910c1969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:16.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9109d9b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c101cb0 0x7f910c1969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42752/0 (socket says 172.21.15.118:42752) 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9109d9b700 1 -- 172.21.15.118:0/3505080158 learned_addr learned my addr 172.21.15.118:0/3505080158 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f910a59c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f910c100ab0 0x7f910c1964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9109d9b700 1 -- 172.21.15.118:0/3505080158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f910c100ab0 msgr2=0x7f910c1964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9109d9b700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f910c100ab0 0x7f910c1964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9109d9b700 1 -- 172.21.15.118:0/3505080158 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc0095f0 con 0x7f910c101cb0 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f910a59c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f910c100ab0 0x7f910c1964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f9109d9b700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c101cb0 0x7f910c1969e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f910000c8a0 tx=0x7f910000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.931+0000 7f90fb7fe700 1 -- 172.21.15.118:0/3505080158 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f910000cea0 con 0x7f910c101cb0 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.932+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c19bba0 con 0x7f910c101cb0 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.932+0000 7f90fb7fe700 1 -- 172.21.15.118:0/3505080158 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9100004830 con 0x7f910c101cb0 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.932+0000 7f90fb7fe700 1 -- 172.21.15.118:0/3505080158 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f91000055b0 con 0x7f910c101cb0 2024-03-12T03:23:16.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.932+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c19c170 con 0x7f910c101cb0 2024-03-12T03:23:16.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.932+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910c190630 con 0x7f910c101cb0 2024-03-12T03:23:16.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.933+0000 7f90fb7fe700 1 -- 172.21.15.118:0/3505080158 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f91000049a0 con 0x7f910c101cb0 2024-03-12T03:23:16.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.933+0000 7f90fb7fe700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f90f406c890 0x7f90f406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:16.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.933+0000 7f90fb7fe700 1 -- 172.21.15.118:0/3505080158 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f910008b980 con 0x7f910c101cb0 2024-03-12T03:23:16.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.933+0000 7f910a59c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f90f406c890 0x7f90f406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:16.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.934+0000 7f910a59c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f90f406c890 0x7f90f406ed50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f90fc006010 tx=0x7f90fc00b540 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:16.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:16.938+0000 7f90fb7fe700 1 -- 172.21.15.118:0/3505080158 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f91000599c0 con 0x7f910c101cb0 2024-03-12T03:23:17.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.134+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f910c04ea80 con 0x7f910c101cb0 2024-03-12T03:23:17.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.135+0000 7f90fb7fe700 1 -- 172.21.15.118:0/3505080158 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f910001d0e0 con 0x7f910c101cb0 2024-03-12T03:23:17.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.137+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f90f406c890 msgr2=0x7f90f406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:17.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.137+0000 7f9110d2c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f90f406c890 0x7f90f406ed50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f90fc006010 tx=0x7f90fc00b540 comp rx=0 tx=0).stop 2024-03-12T03:23:17.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.137+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c101cb0 msgr2=0x7f910c1969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:17.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.137+0000 7f9110d2c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c101cb0 0x7f910c1969e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f910000c8a0 tx=0x7f910000cc60 comp rx=0 tx=0).stop 2024-03-12T03:23:17.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.138+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 shutdown_connections 2024-03-12T03:23:17.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.138+0000 7f9110d2c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f90f406c890 0x7f90f406ed50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:17.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.138+0000 7f9110d2c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f910c100ab0 0x7f910c1964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:17.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.138+0000 7f9110d2c700 1 --2- 172.21.15.118:0/3505080158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f910c101cb0 0x7f910c1969e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:17.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.138+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 >> 172.21.15.118:0/3505080158 conn(0x7f910c0fc090 msgr2=0x7f910c105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:17.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.138+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 shutdown_connections 2024-03-12T03:23:17.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:17.138+0000 7f9110d2c700 1 -- 172.21.15.118:0/3505080158 wait complete. 2024-03-12T03:23:17.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-12T03:23:17.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:17 smithi118 ceph-mon[29160]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:23:17.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:17 smithi177 ceph-mon[37140]: pgmap v199: 65 pgs: 65 active+clean; 581 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:23:18.019 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:18.768 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:18.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:19.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 -- 172.21.15.118:0/561635642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 msgr2=0x7f7fa00ff520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:19.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/561635642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa00ff520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000aa20 tx=0x7f7f9000ad30 comp rx=0 tx=0).stop 2024-03-12T03:23:19.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 -- 172.21.15.118:0/561635642 shutdown_connections 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/561635642 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fa0100300 0x7f7fa0100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/561635642 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa00ff520 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 -- 172.21.15.118:0/561635642 >> 172.21.15.118:0/561635642 conn(0x7f7fa00fa6a0 msgr2=0x7f7fa00fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 -- 172.21.15.118:0/561635642 shutdown_connections 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.592+0000 7f7fa72f8700 1 -- 172.21.15.118:0/561635642 wait complete. 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa72f8700 1 Processor -- start 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa72f8700 1 -- start start 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa72f8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa0194340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa72f8700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fa0100300 0x7f7fa0194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa72f8700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa0194ea0 con 0x7f7fa00ff100 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa72f8700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa0194fe0 con 0x7f7fa0100300 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa5094700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa0194340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa5094700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa0194340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42762/0 (socket says 172.21.15.118:42762) 2024-03-12T03:23:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.593+0000 7f7fa5094700 1 -- 172.21.15.118:0/1542781657 learned_addr learned my addr 172.21.15.118:0/1542781657 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7fa4893700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fa0100300 0x7f7fa0194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7fa5094700 1 -- 172.21.15.118:0/1542781657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fa0100300 msgr2=0x7f7fa0194880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7fa5094700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fa0100300 0x7f7fa0194880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7fa5094700 1 -- 172.21.15.118:0/1542781657 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9000a700 con 0x7f7fa00ff100 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7fa5094700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa0194340 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7f90005b40 tx=0x7f7f90004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7f967fc700 1 -- 172.21.15.118:0/1542781657 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f9001d650 con 0x7f7fa00ff100 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7f967fc700 1 -- 172.21.15.118:0/1542781657 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f90004e00 con 0x7f7fa00ff100 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7f967fc700 1 -- 172.21.15.118:0/1542781657 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7f9000ecb0 con 0x7f7fa00ff100 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa0199990 con 0x7f7fa00ff100 2024-03-12T03:23:19.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.594+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa0199e80 con 0x7f7fa00ff100 2024-03-12T03:23:19.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.595+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa018e410 con 0x7f7fa00ff100 2024-03-12T03:23:19.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.595+0000 7f7f967fc700 1 -- 172.21.15.118:0/1542781657 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7f9000e450 con 0x7f7fa00ff100 2024-03-12T03:23:19.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.596+0000 7f7f967fc700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7f8c06c880 0x7f7f8c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:19.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.596+0000 7f7f967fc700 1 -- 172.21.15.118:0/1542781657 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7f90012070 con 0x7f7fa00ff100 2024-03-12T03:23:19.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.596+0000 7f7fa4893700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7f8c06c880 0x7f7f8c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:19.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.596+0000 7f7fa4893700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7f8c06c880 0x7f7f8c06ed40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c005950 tx=0x7f7f9c0058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:19.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.600+0000 7f7f967fc700 1 -- 172.21.15.118:0/1542781657 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7f9005e030 con 0x7f7fa00ff100 2024-03-12T03:23:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.796+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f7fa00623e0 con 0x7f7fa00ff100 2024-03-12T03:23:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.796+0000 7f7f967fc700 1 -- 172.21.15.118:0/1542781657 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f7fa00623e0 con 0x7f7fa00ff100 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7f8c06c880 msgr2=0x7f7f8c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7f8c06c880 0x7f7f8c06ed40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c005950 tx=0x7f7f9c0058e0 comp rx=0 tx=0).stop 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 msgr2=0x7f7fa0194340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa0194340 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7f90005b40 tx=0x7f7f90004950 comp rx=0 tx=0).stop 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 shutdown_connections 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7f8c06c880 0x7f7f8c06ed40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fa00ff100 0x7f7fa0194340 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 --2- 172.21.15.118:0/1542781657 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7fa0100300 0x7f7fa0194880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 >> 172.21.15.118:0/1542781657 conn(0x7f7fa00fa6a0 msgr2=0x7f7fa0103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 shutdown_connections 2024-03-12T03:23:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:19.798+0000 7f7fa72f8700 1 -- 172.21.15.118:0/1542781657 wait complete. 2024-03-12T03:23:19.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:19 smithi177 ceph-mon[37140]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-12T03:23:20.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:19 smithi118 ceph-mon[29160]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-12T03:23:20.431 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-12T03:23:20.636 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:21.918 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:21 smithi118 ceph-mon[29160]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:23:21.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:21 smithi177 ceph-mon[37140]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 -- 172.21.15.118:0/928052273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 msgr2=0x7fdbb81021d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 --2- 172.21.15.118:0/928052273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81021d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdba8009910 tx=0x7fdba8009c20 comp rx=0 tx=0).stop 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 -- 172.21.15.118:0/928052273 shutdown_connections 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 --2- 172.21.15.118:0/928052273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81021d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 --2- 172.21.15.118:0/928052273 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbb80fed90 0x7fdbb80ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 -- 172.21.15.118:0/928052273 >> 172.21.15.118:0/928052273 conn(0x7fdbb80fa6d0 msgr2=0x7fdbb80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 -- 172.21.15.118:0/928052273 shutdown_connections 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.264+0000 7fdbbff71700 1 -- 172.21.15.118:0/928052273 wait complete. 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbff71700 1 Processor -- start 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbff71700 1 -- start start 2024-03-12T03:23:22.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbff71700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbb80fed90 0x7fdbb81964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbff71700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbff71700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8197010 con 0x7fdbb8101d70 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbff71700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb8197150 con 0x7fdbb80fed90 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbd50c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbd50c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42776/0 (socket says 172.21.15.118:42776) 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbd50c700 1 -- 172.21.15.118:0/4195216771 learned_addr learned my addr 172.21.15.118:0/4195216771 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.265+0000 7fdbbdd0d700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbb80fed90 0x7fdbb81964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbbd50c700 1 -- 172.21.15.118:0/4195216771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbb80fed90 msgr2=0x7fdbb81964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbbd50c700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbb80fed90 0x7fdbb81964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbbd50c700 1 -- 172.21.15.118:0/4195216771 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba80095f0 con 0x7fdbb8101d70 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbbd50c700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81969f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdba80092e0 tx=0x7fdba800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbaeffd700 1 -- 172.21.15.118:0/4195216771 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdba801d070 con 0x7fdbb8101d70 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbaeffd700 1 -- 172.21.15.118:0/4195216771 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdba800f460 con 0x7fdbb8101d70 2024-03-12T03:23:22.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb819bb00 con 0x7fdbb8101d70 2024-03-12T03:23:22.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbaeffd700 1 -- 172.21.15.118:0/4195216771 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdba8021640 con 0x7fdbb8101d70 2024-03-12T03:23:22.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.266+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb819c070 con 0x7fdbb8101d70 2024-03-12T03:23:22.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.267+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb8190650 con 0x7fdbb8101d70 2024-03-12T03:23:22.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.267+0000 7fdbaeffd700 1 -- 172.21.15.118:0/4195216771 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fdba800fad0 con 0x7fdbb8101d70 2024-03-12T03:23:22.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.268+0000 7fdbaeffd700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdba406c880 0x7fdba406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:22.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.268+0000 7fdbaeffd700 1 -- 172.21.15.118:0/4195216771 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fdba808cd60 con 0x7fdbb8101d70 2024-03-12T03:23:22.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.268+0000 7fdbbdd0d700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdba406c880 0x7fdba406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:22.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.268+0000 7fdbbdd0d700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdba406c880 0x7fdba406ed40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4005950 tx=0x7fdbb40058e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:22.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.272+0000 7fdbaeffd700 1 -- 172.21.15.118:0/4195216771 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fdba805aed0 con 0x7fdbb8101d70 2024-03-12T03:23:22.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.464+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7fdbb802cf60 con 0x7fdbb8101d70 2024-03-12T03:23:22.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.465+0000 7fdbaeffd700 1 -- 172.21.15.118:0/4195216771 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7fdba805aa60 con 0x7fdbb8101d70 2024-03-12T03:23:22.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdba406c880 msgr2=0x7fdba406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:22.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdba406c880 0x7fdba406ed40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4005950 tx=0x7fdbb40058e0 comp rx=0 tx=0).stop 2024-03-12T03:23:22.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 msgr2=0x7fdbb81969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81969f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdba80092e0 tx=0x7fdba800be20 comp rx=0 tx=0).stop 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 shutdown_connections 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fdba406c880 0x7fdba406ed40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fdbb80fed90 0x7fdbb81964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 --2- 172.21.15.118:0/4195216771 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdbb8101d70 0x7fdbb81969f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 >> 172.21.15.118:0/4195216771 conn(0x7fdbb80fa6d0 msgr2=0x7fdbb81052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 shutdown_connections 2024-03-12T03:23:22.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:22.467+0000 7fdbbff71700 1 -- 172.21.15.118:0/4195216771 wait complete. 2024-03-12T03:23:23.185 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-12T03:23:23.391 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:23.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:23 smithi118 ceph-mon[29160]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:23:23.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:23 smithi177 ceph-mon[37140]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-12T03:23:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 -- 172.21.15.118:0/626486702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 msgr2=0x7fcba8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 --2- 172.21.15.118:0/626486702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8100ed0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009920 tx=0x7fcba0009c30 comp rx=0 tx=0).stop 2024-03-12T03:23:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 -- 172.21.15.118:0/626486702 shutdown_connections 2024-03-12T03:23:25.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 --2- 172.21.15.118:0/626486702 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba8101cb0 0x7fcba8102110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 --2- 172.21.15.118:0/626486702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8100ed0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 -- 172.21.15.118:0/626486702 >> 172.21.15.118:0/626486702 conn(0x7fcba80fc050 msgr2=0x7fcba80fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 -- 172.21.15.118:0/626486702 shutdown_connections 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.120+0000 7fcbad202700 1 -- 172.21.15.118:0/626486702 wait complete. 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcbad202700 1 Processor -- start 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcbad202700 1 -- start start 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcbad202700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcbad202700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba8101cb0 0x7fcba8196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcbad202700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba8197160 con 0x7fcba8100ab0 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcbad202700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba81972a0 con 0x7fcba8101cb0 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcba6d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcba6d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46292/0 (socket says 172.21.15.118:46292) 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcba6d9d700 1 -- 172.21.15.118:0/2252276128 learned_addr learned my addr 172.21.15.118:0/2252276128 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:25.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.121+0000 7fcba659c700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba8101cb0 0x7fcba8196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcba6d9d700 1 -- 172.21.15.118:0/2252276128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba8101cb0 msgr2=0x7fcba8196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcba6d9d700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba8101cb0 0x7fcba8196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcba6d9d700 1 -- 172.21.15.118:0/2252276128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0009580 con 0x7fcba8100ab0 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcba659c700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba8101cb0 0x7fcba8196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcba6d9d700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8196600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcba0004610 tx=0x7fcba00046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcb8ffff700 1 -- 172.21.15.118:0/2252276128 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba001d070 con 0x7fcba8100ab0 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcb8ffff700 1 -- 172.21.15.118:0/2252276128 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcba000f460 con 0x7fcba8100ab0 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba819bc50 con 0x7fcba8100ab0 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcb8ffff700 1 -- 172.21.15.118:0/2252276128 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcba0021600 con 0x7fcba8100ab0 2024-03-12T03:23:25.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.122+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba819c140 con 0x7fcba8100ab0 2024-03-12T03:23:25.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.123+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba81906b0 con 0x7fcba8100ab0 2024-03-12T03:23:25.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.123+0000 7fcb8ffff700 1 -- 172.21.15.118:0/2252276128 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fcba000fa50 con 0x7fcba8100ab0 2024-03-12T03:23:25.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.124+0000 7fcb8ffff700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb9006c890 0x7fcb9006ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:25.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.124+0000 7fcba659c700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb9006c890 0x7fcb9006ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:25.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.124+0000 7fcb8ffff700 1 -- 172.21.15.118:0/2252276128 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fcba008cf40 con 0x7fcba8100ab0 2024-03-12T03:23:25.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.124+0000 7fcba659c700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb9006c890 0x7fcb9006ed50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0099d0 tx=0x7fcb9c008040 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:25.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.129+0000 7fcb8ffff700 1 -- 172.21.15.118:0/2252276128 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fcba00c0960 con 0x7fcba8100ab0 2024-03-12T03:23:25.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:24 smithi177 ceph-mon[37140]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-12T03:23:25.288 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:24 smithi118 ceph-mon[29160]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-12T03:23:25.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:25.343+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fcba80612a0 con 0x7fcb9006c890 2024-03-12T03:23:26.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:25 smithi177 ceph-mon[37140]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:26.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:25 smithi118 ceph-mon[29160]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:23:27.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:26 smithi177 ceph-mon[37140]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-12T03:23:27.310 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:26 smithi118 ceph-mon[29160]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.6 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-12T03:23:29.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:29 smithi177 ceph-mon[37140]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-12T03:23:30.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:29 smithi118 ceph-mon[29160]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-12T03:23:31.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:31 smithi118 ceph-mon[29160]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:23:31.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:31 smithi177 ceph-mon[37140]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:23:33.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:33 smithi177 ceph-mon[37140]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:23:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:33 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:34.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:33 smithi118 ceph-mon[29160]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:23:34.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:33 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:35.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:34 smithi177 ceph-mon[37140]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:23:35.310 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:34 smithi118 ceph-mon[29160]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:23:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:37 smithi177 ceph-mon[37140]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:23:38.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:37 smithi118 ceph-mon[29160]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:23:39.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:39 smithi177 ceph-mon[37140]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:23:40.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:39 smithi118 ceph-mon[29160]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:23:41.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:41 smithi177 ceph-mon[37140]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-12T03:23:42.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:41 smithi118 ceph-mon[29160]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-12T03:23:43.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:43 smithi177 ceph-mon[37140]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:23:44.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:43 smithi118 ceph-mon[29160]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:23:45.167 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:44 smithi118 ceph-mon[29160]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:23:45.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:44 smithi177 ceph-mon[37140]: pgmap v213: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:23:47.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:47 smithi177 ceph-mon[37140]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:23:48.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:47 smithi118 ceph-mon[29160]: pgmap v214: 65 pgs: 65 active+clean; 586 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:23:48.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:48.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:23:49.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:49 smithi177 ceph-mon[37140]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:23:50.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:49 smithi118 ceph-mon[29160]: pgmap v215: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:23:51.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:51 smithi177 ceph-mon[37140]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:23:52.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:51 smithi118 ceph-mon[29160]: pgmap v216: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:23:53.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:53 smithi177 ceph-mon[37140]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:23:54.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:53 smithi118 ceph-mon[29160]: pgmap v217: 65 pgs: 65 active+clean; 588 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:23:55.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:54 smithi177 ceph-mon[37140]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-12T03:23:55.310 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:54 smithi118 ceph-mon[29160]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-03-12T03:23:56.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.506+0000 7fcb8ffff700 1 -- 172.21.15.118:0/2252276128 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7fcba80612a0 con 0x7fcb9006c890 2024-03-12T03:23:56.509 INFO:teuthology.orchestra.run.smithi118.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:23:56.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb9006c890 msgr2=0x7fcb9006ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:56.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb9006c890 0x7fcb9006ed50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0099d0 tx=0x7fcb9c008040 comp rx=0 tx=0).stop 2024-03-12T03:23:56.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 msgr2=0x7fcba8196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:56.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8196600 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcba0004610 tx=0x7fcba00046f0 comp rx=0 tx=0).stop 2024-03-12T03:23:56.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 shutdown_connections 2024-03-12T03:23:56.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fcb9006c890 0x7fcb9006ed50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:56.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcba8100ab0 0x7fcba8196600 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:56.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 --2- 172.21.15.118:0/2252276128 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fcba8101cb0 0x7fcba8196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:56.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 >> 172.21.15.118:0/2252276128 conn(0x7fcba80fc050 msgr2=0x7fcba8105300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:56.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 shutdown_connections 2024-03-12T03:23:56.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:56.508+0000 7fcbad202700 1 -- 172.21.15.118:0/2252276128 wait complete. 2024-03-12T03:23:57.170 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T03:23:57.426 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:23:57.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:57 smithi118 ceph-mon[29160]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:23:57.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:57.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:57.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:57 smithi118 ceph-mon[29160]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-12T03:23:57.816 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:57.817 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:57.817 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:57 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:57.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:57 smithi177 ceph-mon[37140]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:23:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:23:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:57 smithi177 ceph-mon[37140]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-12T03:23:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:23:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:23:57.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:57 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:23:58.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:58 smithi118 ceph-mon[29160]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:23:58.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:58 smithi177 ceph-mon[37140]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:23:59.743 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:23:59 smithi118 ceph-mon[29160]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 -- 172.21.15.118:0/722200314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4103ea0 msgr2=0x7fbdb4106290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 --2- 172.21.15.118:0/722200314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4103ea0 0x7fbdb4106290 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009910 tx=0x7fbda8009c20 comp rx=0 tx=0).stop 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 -- 172.21.15.118:0/722200314 shutdown_connections 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 --2- 172.21.15.118:0/722200314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4103ea0 0x7fbdb4106290 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 --2- 172.21.15.118:0/722200314 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdb4101570 0x7fbdb4103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 -- 172.21.15.118:0/722200314 >> 172.21.15.118:0/722200314 conn(0x7fbdb40faf60 msgr2=0x7fbdb40fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 -- 172.21.15.118:0/722200314 shutdown_connections 2024-03-12T03:23:59.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.752+0000 7fbdb9154700 1 -- 172.21.15.118:0/722200314 wait complete. 2024-03-12T03:23:59.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.753+0000 7fbdb9154700 1 Processor -- start 2024-03-12T03:23:59.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.753+0000 7fbdb9154700 1 -- start start 2024-03-12T03:23:59.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb9154700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4101570 0x7fbdb4196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:59.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb9154700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdb4103ea0 0x7fbdb41969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:59.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb9154700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4196fd0 con 0x7fbdb4101570 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb9154700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb4197110 con 0x7fbdb4103ea0 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb2d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4101570 0x7fbdb4196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb2d9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4101570 0x7fbdb4196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60404/0 (socket says 172.21.15.118:60404) 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb2d9d700 1 -- 172.21.15.118:0/3187177070 learned_addr learned my addr 172.21.15.118:0/3187177070 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb259c700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdb4103ea0 0x7fbdb41969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb2d9d700 1 -- 172.21.15.118:0/3187177070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdb4103ea0 msgr2=0x7fbdb41969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb2d9d700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdb4103ea0 0x7fbdb41969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:23:59.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb2d9d700 1 -- 172.21.15.118:0/3187177070 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda80095f0 con 0x7fbdb4101570 2024-03-12T03:23:59.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.754+0000 7fbdb2d9d700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4101570 0x7fbdb4196470 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbda400d900 tx=0x7fbda400dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:59.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.755+0000 7fbd9bfff700 1 -- 172.21.15.118:0/3187177070 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda40041f0 con 0x7fbdb4101570 2024-03-12T03:23:59.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.755+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdb419bb70 con 0x7fbdb4101570 2024-03-12T03:23:59.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.755+0000 7fbd9bfff700 1 -- 172.21.15.118:0/3187177070 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbda4004cf0 con 0x7fbdb4101570 2024-03-12T03:23:59.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.755+0000 7fbd9bfff700 1 -- 172.21.15.118:0/3187177070 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbda400b750 con 0x7fbdb4101570 2024-03-12T03:23:59.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.755+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb40ff670 con 0x7fbdb4101570 2024-03-12T03:23:59.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.756+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb40623e0 con 0x7fbdb4101570 2024-03-12T03:23:59.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.756+0000 7fbd9bfff700 1 -- 172.21.15.118:0/3187177070 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fbda4004830 con 0x7fbdb4101570 2024-03-12T03:23:59.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.757+0000 7fbd9bfff700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fbd9c070c90 0x7fbd9c073150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:23:59.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.757+0000 7fbd9bfff700 1 -- 172.21.15.118:0/3187177070 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fbda408bc60 con 0x7fbdb4101570 2024-03-12T03:23:59.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.757+0000 7fbdb259c700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fbd9c070c90 0x7fbd9c073150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:23:59.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.758+0000 7fbdb259c700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fbd9c070c90 0x7fbd9c073150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009fd0 tx=0x7fbda8005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:23:59.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:23:59.762+0000 7fbd9bfff700 1 -- 172.21.15.118:0/3187177070 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fbda4059ca0 con 0x7fbdb4101570 2024-03-12T03:23:59.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:23:59 smithi177 ceph-mon[37140]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-12T03:24:00.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.158+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb4105dd0 con 0x7fbd9c070c90 2024-03-12T03:24:00.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.159+0000 7fbd9bfff700 1 -- 172.21.15.118:0/3187177070 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fbdb4105dd0 con 0x7fbd9c070c90 2024-03-12T03:24:00.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fbd9c070c90 msgr2=0x7fbd9c073150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fbd9c070c90 0x7fbd9c073150 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009fd0 tx=0x7fbda8005960 comp rx=0 tx=0).stop 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4101570 msgr2=0x7fbdb4196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4101570 0x7fbdb4196470 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbda400d900 tx=0x7fbda400dc10 comp rx=0 tx=0).stop 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 shutdown_connections 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fbd9c070c90 0x7fbd9c073150 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbdb4101570 0x7fbdb4196470 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 --2- 172.21.15.118:0/3187177070 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fbdb4103ea0 0x7fbdb41969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 >> 172.21.15.118:0/3187177070 conn(0x7fbdb40faf60 msgr2=0x7fbdb40fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 shutdown_connections 2024-03-12T03:24:00.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.162+0000 7fbdb9154700 1 -- 172.21.15.118:0/3187177070 wait complete. 2024-03-12T03:24:00.175 INFO:teuthology.orchestra.run.smithi118.stdout:true 2024-03-12T03:24:00.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.269+0000 7fe6721df700 1 -- 172.21.15.118:0/3773782003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c101d90 msgr2=0x7fe66c1021f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.269+0000 7fe6721df700 1 --2- 172.21.15.118:0/3773782003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c101d90 0x7fe66c1021f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe660009910 tx=0x7fe660009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:00.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.270+0000 7fe6721df700 1 -- 172.21.15.118:0/3773782003 shutdown_connections 2024-03-12T03:24:00.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.270+0000 7fe6721df700 1 --2- 172.21.15.118:0/3773782003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c101d90 0x7fe66c1021f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.270+0000 7fe6721df700 1 --2- 172.21.15.118:0/3773782003 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe66c0fedb0 0x7fe66c0ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.270+0000 7fe6721df700 1 -- 172.21.15.118:0/3773782003 >> 172.21.15.118:0/3773782003 conn(0x7fe66c0fa6d0 msgr2=0x7fe66c0fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:00.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.270+0000 7fe6721df700 1 -- 172.21.15.118:0/3773782003 shutdown_connections 2024-03-12T03:24:00.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.270+0000 7fe6721df700 1 -- 172.21.15.118:0/3773782003 wait complete. 2024-03-12T03:24:00.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.271+0000 7fe6721df700 1 Processor -- start 2024-03-12T03:24:00.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.271+0000 7fe6721df700 1 -- start start 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe6721df700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c0fedb0 0x7fe66c101260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe6721df700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe66c101d90 0x7fe66c101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe6721df700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c0ffc60 con 0x7fe66c0fedb0 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe6721df700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c0ffdd0 con 0x7fe66c101d90 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe66b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c0fedb0 0x7fe66c101260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe66b7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c0fedb0 0x7fe66c101260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60426/0 (socket says 172.21.15.118:60426) 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe66b7fe700 1 -- 172.21.15.118:0/2680513928 learned_addr learned my addr 172.21.15.118:0/2680513928 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.272+0000 7fe66affd700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe66c101d90 0x7fe66c101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:00.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe66b7fe700 1 -- 172.21.15.118:0/2680513928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe66c101d90 msgr2=0x7fe66c101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe66b7fe700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe66c101d90 0x7fe66c101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe66b7fe700 1 -- 172.21.15.118:0/2680513928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6600095f0 con 0x7fe66c0fedb0 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe66b7fe700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c0fedb0 0x7fe66c101260 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00e8f0 tx=0x7fe65c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe668ff9700 1 -- 172.21.15.118:0/2680513928 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65c009790 con 0x7fe66c0fedb0 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe668ff9700 1 -- 172.21.15.118:0/2680513928 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe65c00c810 con 0x7fe66c0fedb0 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe668ff9700 1 -- 172.21.15.118:0/2680513928 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe65c003d20 con 0x7fe66c0fedb0 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.273+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c1000b0 con 0x7fe66c0fedb0 2024-03-12T03:24:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.274+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c100680 con 0x7fe66c0fedb0 2024-03-12T03:24:00.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.275+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66c100950 con 0x7fe66c0fedb0 2024-03-12T03:24:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.276+0000 7fe668ff9700 1 -- 172.21.15.118:0/2680513928 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fe65c0041f0 con 0x7fe66c0fedb0 2024-03-12T03:24:00.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.277+0000 7fe668ff9700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe65406c890 0x7fe65406ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:00.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.277+0000 7fe668ff9700 1 -- 172.21.15.118:0/2680513928 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fe65c014070 con 0x7fe66c0fedb0 2024-03-12T03:24:00.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.277+0000 7fe66affd700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe65406c890 0x7fe65406ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:00.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.278+0000 7fe66affd700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe65406c890 0x7fe65406ed50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe66c100fe0 tx=0x7fe660000bc0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:00.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.285+0000 7fe668ff9700 1 -- 172.21.15.118:0/2680513928 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fe65c056500 con 0x7fe66c0fedb0 2024-03-12T03:24:00.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.640+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe66c02cf90 con 0x7fe65406c890 2024-03-12T03:24:00.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.641+0000 7fe668ff9700 1 -- 172.21.15.118:0/2680513928 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fe66c02cf90 con 0x7fe65406c890 2024-03-12T03:24:00.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe65406c890 msgr2=0x7fe65406ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe65406c890 0x7fe65406ed50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe66c100fe0 tx=0x7fe660000bc0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c0fedb0 msgr2=0x7fe66c101260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c0fedb0 0x7fe66c101260 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe65c00e8f0 tx=0x7fe65c00ec00 comp rx=0 tx=0).stop 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 shutdown_connections 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fe65406c890 0x7fe65406ed50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe66c0fedb0 0x7fe66c101260 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 --2- 172.21.15.118:0/2680513928 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fe66c101d90 0x7fe66c101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.644+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 >> 172.21.15.118:0/2680513928 conn(0x7fe66c0fa6d0 msgr2=0x7fe66c105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.645+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 shutdown_connections 2024-03-12T03:24:00.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.645+0000 7fe6721df700 1 -- 172.21.15.118:0/2680513928 wait complete. 2024-03-12T03:24:00.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.802+0000 7f2b823ac700 1 -- 172.21.15.118:0/4070455298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 msgr2=0x7f2b7c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.802+0000 7f2b823ac700 1 --2- 172.21.15.118:0/4070455298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 0x7f2b7c101020 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009910 tx=0x7f2b70009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:00.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.803+0000 7f2b823ac700 1 -- 172.21.15.118:0/4070455298 shutdown_connections 2024-03-12T03:24:00.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.803+0000 7f2b823ac700 1 --2- 172.21.15.118:0/4070455298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 0x7f2b7c101020 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.803+0000 7f2b823ac700 1 --2- 172.21.15.118:0/4070455298 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 0x7f2b7c0ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.803+0000 7f2b823ac700 1 -- 172.21.15.118:0/4070455298 >> 172.21.15.118:0/4070455298 conn(0x7f2b7c0faf60 msgr2=0x7f2b7c0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:00.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.803+0000 7f2b823ac700 1 -- 172.21.15.118:0/4070455298 shutdown_connections 2024-03-12T03:24:00.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.803+0000 7f2b823ac700 1 -- 172.21.15.118:0/4070455298 wait complete. 2024-03-12T03:24:00.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.804+0000 7f2b823ac700 1 Processor -- start 2024-03-12T03:24:00.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.804+0000 7f2b823ac700 1 -- start start 2024-03-12T03:24:00.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.804+0000 7f2b823ac700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 0x7f2b7c1953a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:00.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.804+0000 7f2b823ac700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 0x7f2b7c1958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:00.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.804+0000 7f2b823ac700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c195f00 con 0x7f2b7c100bc0 2024-03-12T03:24:00.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.804+0000 7f2b823ac700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c196040 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 0x7f2b7c1953a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7bfff700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 0x7f2b7c1953a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.118:35638/0 (socket says 172.21.15.118:35638) 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7bfff700 1 -- 172.21.15.118:0/3562576650 learned_addr learned my addr 172.21.15.118:0/3562576650 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7b7fe700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 0x7f2b7c1958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7bfff700 1 -- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 msgr2=0x7f2b7c1958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7bfff700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 0x7f2b7c1958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7bfff700 1 -- 172.21.15.118:0/3562576650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b700095f0 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7b7fe700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 0x7f2b7c1958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.805+0000 7f2b7bfff700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 0x7f2b7c1953a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00e8f0 tx=0x7f2b6c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:00.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.806+0000 7f2b797fa700 1 -- 172.21.15.118:0/3562576650 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6c009790 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.806+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7c19aaa0 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.806+0000 7f2b797fa700 1 -- 172.21.15.118:0/3562576650 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b6c0044e0 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.806+0000 7f2b797fa700 1 -- 172.21.15.118:0/3562576650 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b6c003e00 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.806+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c19b070 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.807+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b7c18f540 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.808+0000 7f2b797fa700 1 -- 172.21.15.118:0/3562576650 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f2b6c00cba0 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.809+0000 7f2b797fa700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f2b6406c880 0x7f2b6406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:00.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.809+0000 7f2b797fa700 1 -- 172.21.15.118:0/3562576650 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f2b6c014070 con 0x7f2b7c0ff9c0 2024-03-12T03:24:00.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.809+0000 7f2b7b7fe700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f2b6406c880 0x7f2b6406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:00.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.810+0000 7f2b7b7fe700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f2b6406c880 0x7f2b6406ed40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009fd0 tx=0x7f2b70005fb0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:00.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:00.818+0000 7f2b797fa700 1 -- 172.21.15.118:0/3562576650 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f2b6c059910 con 0x7f2b7c0ff9c0 2024-03-12T03:24:01.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.064+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b7c0611f0 con 0x7f2b6406c880 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.083+0000 7f2b797fa700 1 -- 172.21.15.118:0/3562576650 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3792 (secure 0 0 0) 0x7f2b7c0611f0 con 0x7f2b6406c880 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stdout:alertmanager.smithi118 smithi118 *:9093,9094 running (5m) 52s ago 6m 24.9M - 0.25.0 c8568f914cd2 ab6c8c340db8 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-exporter.smithi118 smithi118 running (7m) 52s ago 7m 7507k - 18.2.2-751-g62a11b3c fa4cc2e02803 dab7493ee414 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-exporter.smithi177 smithi177 running (6m) 56s ago 6m 7453k - 18.2.2-751-g62a11b3c fa4cc2e02803 56905fbf68ab 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stdout:crash.smithi118 smithi118 running (7m) 52s ago 7m 6660k - 18.2.2-751-g62a11b3c fa4cc2e02803 848a90530bcc 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stdout:crash.smithi177 smithi177 running (5m) 56s ago 5m 6660k - 18.2.2-751-g62a11b3c fa4cc2e02803 73a879109cfd 2024-03-12T03:24:01.086 INFO:teuthology.orchestra.run.smithi118.stdout:grafana.smithi118 smithi118 *:3000 running (5m) 52s ago 6m 83.8M - 9.4.7 954c08fa6188 f56cb9df4974 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi118.oysrod smithi118 running (72s) 52s ago 72s 18.1M - 18.2.2-751-g62a11b3c fa4cc2e02803 0579203a308a 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi118.wouiai smithi118 running (80s) 52s ago 78s 13.0M - 18.2.2-751-g62a11b3c fa4cc2e02803 47ff6106211e 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi177.azybll smithi177 running (69s) 56s ago 69s 16.6M - 18.2.2-751-g62a11b3c fa4cc2e02803 237998f5045b 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi177.kbzhid smithi177 running (76s) 56s ago 76s 16.0M - 18.2.2-751-g62a11b3c fa4cc2e02803 71614295c01b 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.smithi118.zzzuvl smithi118 *:9283,8765,8443 running (8m) 52s ago 8m 497M - 18.2.2-751-g62a11b3c fa4cc2e02803 b31fb006922b 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.smithi177.tmmsuz smithi177 *:8443,9283,8765 running (5m) 56s ago 5m 453M - 18.2.2-751-g62a11b3c fa4cc2e02803 9f6d8c95b283 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mon.smithi118 smithi118 running (8m) 52s ago 8m 45.5M 2048M 18.2.2-751-g62a11b3c fa4cc2e02803 a26ddd75d886 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:mon.smithi177 smithi177 running (5m) 56s ago 5m 36.9M 2048M 18.2.2-751-g62a11b3c fa4cc2e02803 0d8acd4f44b3 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:node-exporter.smithi118 smithi118 *:9100 running (7m) 52s ago 7m 16.3M - 1.5.0 0da6a335fe13 4e2acb3f4a85 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:node-exporter.smithi177 smithi177 *:9100 running (5m) 56s ago 5m 16.8M - 1.5.0 0da6a335fe13 bde9c5ddfe9c 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:osd.0 smithi118 running (5m) 52s ago 5m 50.4M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 9436f155422d 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:osd.1 smithi118 running (4m) 52s ago 4m 48.2M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 38fd187fe41b 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:osd.2 smithi118 running (4m) 52s ago 4m 46.0M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 a5856da80958 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:osd.3 smithi177 running (3m) 56s ago 3m 47.6M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 1a594e7be73e 2024-03-12T03:24:01.087 INFO:teuthology.orchestra.run.smithi118.stdout:osd.4 smithi177 running (3m) 56s ago 3m 47.0M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 a4b1d7721078 2024-03-12T03:24:01.088 INFO:teuthology.orchestra.run.smithi118.stdout:osd.5 smithi177 running (2m) 56s ago 2m 43.5M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 fd4d4398ebeb 2024-03-12T03:24:01.088 INFO:teuthology.orchestra.run.smithi118.stdout:prometheus.smithi118 smithi118 *:9095 running (5m) 52s ago 6m 53.6M - 2.43.0 a07b618ecd1d 367c172183f1 2024-03-12T03:24:01.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.085+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f2b6406c880 msgr2=0x7f2b6406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.085+0000 7f2b823ac700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f2b6406c880 0x7f2b6406ed40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009fd0 tx=0x7f2b70005fb0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.085+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 msgr2=0x7f2b7c1953a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.085+0000 7f2b823ac700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 0x7f2b7c1953a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00e8f0 tx=0x7f2b6c00ecb0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.085+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 shutdown_connections 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.086+0000 7f2b823ac700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f2b6406c880 0x7f2b6406ed40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.086+0000 7f2b823ac700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f2b7c0ff9c0 0x7f2b7c1953a0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00e8f0 tx=0x7f2b6c00ecb0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.086+0000 7f2b823ac700 1 --2- 172.21.15.118:0/3562576650 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2b7c100bc0 0x7f2b7c1958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.086+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 >> 172.21.15.118:0/3562576650 conn(0x7f2b7c0faf60 msgr2=0x7f2b7c1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.086+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 shutdown_connections 2024-03-12T03:24:01.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.086+0000 7f2b823ac700 1 -- 172.21.15.118:0/3562576650 wait complete. 2024-03-12T03:24:01.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 -- 172.21.15.118:0/1991971065 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 msgr2=0x7f26781001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 --2- 172.21.15.118:0/1991971065 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f26781001d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f266c009910 tx=0x7f266c009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 -- 172.21.15.118:0/1991971065 shutdown_connections 2024-03-12T03:24:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 --2- 172.21.15.118:0/1991971065 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f26781001d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 --2- 172.21.15.118:0/1991971065 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26780ff3f0 0x7f26780ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 -- 172.21.15.118:0/1991971065 >> 172.21.15.118:0/1991971065 conn(0x7f26780fafb0 msgr2=0x7f26780fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 -- 172.21.15.118:0/1991971065 shutdown_connections 2024-03-12T03:24:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.193+0000 7f267d959700 1 -- 172.21.15.118:0/1991971065 wait complete. 2024-03-12T03:24:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.194+0000 7f267d959700 1 Processor -- start 2024-03-12T03:24:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.194+0000 7f267d959700 1 -- start start 2024-03-12T03:24:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.194+0000 7f267d959700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26780ff3f0 0x7f26781986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.194+0000 7f267d959700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f2678198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.194+0000 7f267d959700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2678199210 con 0x7f26780ffd50 2024-03-12T03:24:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.194+0000 7f267d959700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2678199350 con 0x7f26780ff3f0 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f26767fc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f2678198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f26767fc700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f2678198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60450/0 (socket says 172.21.15.118:60450) 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f26767fc700 1 -- 172.21.15.118:0/1817383729 learned_addr learned my addr 172.21.15.118:0/1817383729 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f2676ffd700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26780ff3f0 0x7f26781986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f26767fc700 1 -- 172.21.15.118:0/1817383729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26780ff3f0 msgr2=0x7f26781986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f26767fc700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26780ff3f0 0x7f26781986b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f26767fc700 1 -- 172.21.15.118:0/1817383729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266c0095f0 con 0x7f26780ffd50 2024-03-12T03:24:01.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f26767fc700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f2678198bf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f266c006010 tx=0x7f266c00bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f267c957700 1 -- 172.21.15.118:0/1817383729 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f266c01d070 con 0x7f26780ffd50 2024-03-12T03:24:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267819dd00 con 0x7f26780ffd50 2024-03-12T03:24:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f267c957700 1 -- 172.21.15.118:0/1817383729 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f266c00f460 con 0x7f26780ffd50 2024-03-12T03:24:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.195+0000 7f267c957700 1 -- 172.21.15.118:0/1817383729 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f266c021620 con 0x7f26780ffd50 2024-03-12T03:24:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.196+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267819e1f0 con 0x7f26780ffd50 2024-03-12T03:24:01.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.197+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2678192870 con 0x7f26780ffd50 2024-03-12T03:24:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.199+0000 7f267c957700 1 -- 172.21.15.118:0/1817383729 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f266c003a60 con 0x7f26780ffd50 2024-03-12T03:24:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.199+0000 7f267c957700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f266006c880 0x7f266006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.199+0000 7f267c957700 1 -- 172.21.15.118:0/1817383729 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f266c08ce40 con 0x7f26780ffd50 2024-03-12T03:24:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.200+0000 7f2676ffd700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f266006c880 0x7f266006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.200+0000 7f2676ffd700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f266006c880 0x7f266006ed40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2668007c60 tx=0x7f26680073d0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:01.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.204+0000 7f267c957700 1 -- 172.21.15.118:0/1817383729 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f266c05ae80 con 0x7f26780ffd50 2024-03-12T03:24:01.452 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (unknown) 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: Upgrade: Need to upgrade myself (mgr.smithi118.zzzuvl) 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:01.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:01 smithi177 ceph-mon[37140]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 on smithi177 2024-03-12T03:24:01.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.502+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f267804ea80 con 0x7f26780ffd50 2024-03-12T03:24:01.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.503+0000 7f267c957700 1 -- 172.21.15.118:0/1817383729 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f266c05aa10 con 0x7f26780ffd50 2024-03-12T03:24:01.505 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "mon": { 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 2 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "mgr": { 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 2 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "osd": { 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 6 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "mds": { 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 4 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "overall": { 2024-03-12T03:24:01.506 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 14 2024-03-12T03:24:01.507 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-12T03:24:01.507 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f266006c880 msgr2=0x7f266006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f266006c880 0x7f266006ed40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2668007c60 tx=0x7f26680073d0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 msgr2=0x7f2678198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f2678198bf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f266c006010 tx=0x7f266c00bd70 comp rx=0 tx=0).stop 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 shutdown_connections 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f266006c880 0x7f266006ed40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f26780ff3f0 0x7f26781986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 --2- 172.21.15.118:0/1817383729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f26780ffd50 0x7f2678198bf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.505+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 >> 172.21.15.118:0/1817383729 conn(0x7f26780fafb0 msgr2=0x7f2678107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:01.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.506+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 shutdown_connections 2024-03-12T03:24:01.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.506+0000 7f267d959700 1 -- 172.21.15.118:0/1817383729 wait complete. 2024-03-12T03:24:01.616 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:01.616 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:01.616 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (unknown) 2024-03-12T03:24:01.616 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:24:01.616 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:01.616 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:01.617 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: Upgrade: Need to upgrade myself (mgr.smithi118.zzzuvl) 2024-03-12T03:24:01.617 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-12T03:24:01.617 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:01.617 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:01 smithi118 ceph-mon[29160]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 on smithi177 2024-03-12T03:24:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.622+0000 7ff0411da700 1 -- 172.21.15.118:0/127580715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 msgr2=0x7ff03c1000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.622+0000 7ff0411da700 1 --2- 172.21.15.118:0/127580715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 0x7ff03c1000b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff030009910 tx=0x7ff030009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.622+0000 7ff0411da700 1 -- 172.21.15.118:0/127580715 shutdown_connections 2024-03-12T03:24:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.622+0000 7ff0411da700 1 --2- 172.21.15.118:0/127580715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 0x7ff03c1000b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.622+0000 7ff0411da700 1 --2- 172.21.15.118:0/127580715 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 0x7ff03c101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.622+0000 7ff0411da700 1 -- 172.21.15.118:0/127580715 >> 172.21.15.118:0/127580715 conn(0x7ff03c0fa6a0 msgr2=0x7ff03c0fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.623+0000 7ff0411da700 1 -- 172.21.15.118:0/127580715 shutdown_connections 2024-03-12T03:24:01.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.623+0000 7ff0411da700 1 -- 172.21.15.118:0/127580715 wait complete. 2024-03-12T03:24:01.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.624+0000 7ff0411da700 1 Processor -- start 2024-03-12T03:24:01.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.624+0000 7ff0411da700 1 -- start start 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.624+0000 7ff0411da700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 0x7ff03c1977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.624+0000 7ff0411da700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 0x7ff03c197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.624+0000 7ff0411da700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c194340 con 0x7ff03c0ffc50 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff03a59c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 0x7ff03c197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff03a59c700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 0x7ff03c197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.177:3300/0 says I am v2:172.21.15.118:35682/0 (socket says 172.21.15.118:35682) 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff03ad9d700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 0x7ff03c1977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff03a59c700 1 -- 172.21.15.118:0/1615416800 learned_addr learned my addr 172.21.15.118:0/1615416800 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c194480 con 0x7ff03c100f10 2024-03-12T03:24:01.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff03a59c700 1 -- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 msgr2=0x7ff03c1977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff03a59c700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 0x7ff03c1977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.625+0000 7ff03a59c700 1 -- 172.21.15.118:0/1615416800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0300095f0 con 0x7ff03c100f10 2024-03-12T03:24:01.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.626+0000 7ff03a59c700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 0x7ff03c197cf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff03000b6b0 tx=0x7ff03000b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:01.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.626+0000 7ff023fff700 1 -- 172.21.15.118:0/1615416800 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff0300052a0 con 0x7ff03c100f10 2024-03-12T03:24:01.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.626+0000 7ff023fff700 1 -- 172.21.15.118:0/1615416800 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff030003680 con 0x7ff03c100f10 2024-03-12T03:24:01.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.626+0000 7ff023fff700 1 -- 172.21.15.118:0/1615416800 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff03001f670 con 0x7ff03c100f10 2024-03-12T03:24:01.629 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.626+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03c194700 con 0x7ff03c100f10 2024-03-12T03:24:01.629 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.627+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03c194bb0 con 0x7ff03c100f10 2024-03-12T03:24:01.630 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.628+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03c18e3f0 con 0x7ff03c100f10 2024-03-12T03:24:01.631 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.629+0000 7ff023fff700 1 -- 172.21.15.118:0/1615416800 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7ff0300037f0 con 0x7ff03c100f10 2024-03-12T03:24:01.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.629+0000 7ff023fff700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff02406c880 0x7ff02406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:01.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.630+0000 7ff023fff700 1 -- 172.21.15.118:0/1615416800 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7ff03008d710 con 0x7ff03c100f10 2024-03-12T03:24:01.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.630+0000 7ff03ad9d700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff02406c880 0x7ff02406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:01.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.630+0000 7ff03ad9d700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff02406c880 0x7ff02406ed40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff03c068180 tx=0x7ff02c009260 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:01.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.638+0000 7ff023fff700 1 -- 172.21.15.118:0/1615416800 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7ff03005b730 con 0x7ff03c100f10 2024-03-12T03:24:01.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.972+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff03c02cfc0 con 0x7ff02406c880 2024-03-12T03:24:01.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.974+0000 7ff023fff700 1 -- 172.21.15.118:0/1615416800 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7ff03c02cfc0 con 0x7ff02406c880 2024-03-12T03:24:01.977 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-12T03:24:01.977 INFO:teuthology.orchestra.run.smithi118.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379", 2024-03-12T03:24:01.977 INFO:teuthology.orchestra.run.smithi118.stdout: "in_progress": true, 2024-03-12T03:24:01.977 INFO:teuthology.orchestra.run.smithi118.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-12T03:24:01.977 INFO:teuthology.orchestra.run.smithi118.stdout: "services_complete": [], 2024-03-12T03:24:01.977 INFO:teuthology.orchestra.run.smithi118.stdout: "progress": "0/2 daemons upgraded", 2024-03-12T03:24:01.978 INFO:teuthology.orchestra.run.smithi118.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 image on host smithi177", 2024-03-12T03:24:01.978 INFO:teuthology.orchestra.run.smithi118.stdout: "is_paused": false 2024-03-12T03:24:01.978 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.977+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff02406c880 msgr2=0x7ff02406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.977+0000 7ff0411da700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff02406c880 0x7ff02406ed40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff03c068180 tx=0x7ff02c009260 comp rx=0 tx=0).stop 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.977+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 msgr2=0x7ff03c197cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.977+0000 7ff0411da700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 0x7ff03c197cf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff03000b6b0 tx=0x7ff03000b6e0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.977+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 shutdown_connections 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.977+0000 7ff0411da700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7ff02406c880 0x7ff02406ed40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.978+0000 7ff0411da700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff03c0ffc50 0x7ff03c1977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.978+0000 7ff0411da700 1 --2- 172.21.15.118:0/1615416800 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7ff03c100f10 0x7ff03c197cf0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.978+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 >> 172.21.15.118:0/1615416800 conn(0x7ff03c0fa6a0 msgr2=0x7ff03c1042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:01.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.978+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 shutdown_connections 2024-03-12T03:24:01.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:01.978+0000 7ff0411da700 1 -- 172.21.15.118:0/1615416800 wait complete. 2024-03-12T03:24:02.682 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:02 smithi118 ceph-mon[29160]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:02.682 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:02 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/1817383729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:24:02.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:02 smithi177 ceph-mon[37140]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:02.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:02 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/1817383729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:24:03.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:03 smithi177 ceph-mon[37140]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:03.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:03 smithi177 ceph-mon[37140]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:24:03.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:03 smithi118 ceph-mon[29160]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:03.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:03 smithi118 ceph-mon[29160]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:24:04.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:04 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:04.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:04 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:05.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:05 smithi177 ceph-mon[37140]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:24:05.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:05 smithi118 ceph-mon[29160]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.4 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-12T03:24:07.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:07 smithi177 ceph-mon[37140]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:24:08.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:07 smithi118 ceph-mon[29160]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:24:09.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:09 smithi177 ceph-mon[37140]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:24:10.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:09 smithi118 ceph-mon[29160]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:24:11.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:11 smithi177 ceph-mon[37140]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:24:11.990 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:11 smithi118 ceph-mon[29160]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:24:13.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:13 smithi177 ceph-mon[37140]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:24:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:13 smithi118 ceph-mon[29160]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:24:15.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:14 smithi177 ceph-mon[37140]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:24:15.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:14 smithi118 ceph-mon[29160]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:24:17.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:17 smithi177 ceph-mon[37140]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:24:18.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:17 smithi118 ceph-mon[29160]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:24:18.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:18 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:18.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:18 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:19.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:19 smithi177 ceph-mon[37140]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:24:20.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:19 smithi118 ceph-mon[29160]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-12T03:24:21.702 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:21 smithi177 ceph-mon[37140]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-12T03:24:22.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:21 smithi118 ceph-mon[29160]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-12T03:24:22.157 INFO:tasks.workunit.client.1.smithi177.stderr:Updating files: 94% (11644/12344) Updating files: 95% (11727/12344) Updating files: 96% (11851/12344) Updating files: 97% (11974/12344) Updating files: 98% (12098/12344) Updating files: 99% (12221/12344) Updating files: 100% (12344/12344) Updating files: 100% (12344/12344), done. 2024-03-12T03:24:22.308 INFO:tasks.workunit.client.1.smithi177.stderr:Note: switching to '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'. 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr: 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr:state without impacting any branches by switching back to a branch. 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr: 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr: 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr: git switch -c 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr: 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr:Or undo this operation with: 2024-03-12T03:24:22.309 INFO:tasks.workunit.client.1.smithi177.stderr: 2024-03-12T03:24:22.310 INFO:tasks.workunit.client.1.smithi177.stderr: git switch - 2024-03-12T03:24:22.310 INFO:tasks.workunit.client.1.smithi177.stderr: 2024-03-12T03:24:22.310 INFO:tasks.workunit.client.1.smithi177.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-12T03:24:22.310 INFO:tasks.workunit.client.1.smithi177.stderr: 2024-03-12T03:24:22.310 INFO:tasks.workunit.client.1.smithi177.stderr:HEAD is now at 1c349451176 Merge pull request #56077 from idryomov/wip-rbd-mirror-bash 2024-03-12T03:24:22.311 DEBUG:teuthology.orchestra.run.smithi177:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-12T03:24:22.371 INFO:tasks.workunit.client.1.smithi177.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-12T03:24:22.373 INFO:tasks.workunit.client.1.smithi177.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-12T03:24:22.373 INFO:tasks.workunit.client.1.smithi177.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-12T03:24:23.148 INFO:tasks.workunit.client.1.smithi177.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-12T03:24:23.211 INFO:tasks.workunit.client.1.smithi177.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-12T03:24:23.252 INFO:tasks.workunit.client.1.smithi177.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-12T03:24:23.254 INFO:tasks.workunit.client.1.smithi177.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-12T03:24:23.254 INFO:tasks.workunit.client.1.smithi177.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-12T03:24:23.295 INFO:tasks.workunit.client.1.smithi177.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-12T03:24:23.298 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:24:23.298 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-12T03:24:23.354 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-12T03:24:23.356 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-12T03:24:23.356 DEBUG:teuthology.orchestra.run.smithi177:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-12T03:24:23.414 INFO:tasks.workunit.client.1.smithi177.stderr:+ mkdir -p fsstress 2024-03-12T03:24:23.416 INFO:tasks.workunit.client.1.smithi177.stderr:+ pushd fsstress 2024-03-12T03:24:23.416 INFO:tasks.workunit.client.1.smithi177.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-12T03:24:23.416 INFO:tasks.workunit.client.1.smithi177.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-12T03:24:23.622 INFO:tasks.workunit.client.0.smithi118.stderr:Note: switching to '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'. 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr:state without impacting any branches by switching back to a branch. 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: git switch -c 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr:Or undo this operation with: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: git switch - 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:24:23.623 INFO:tasks.workunit.client.0.smithi118.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-12T03:24:23.624 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:24:23.624 INFO:tasks.workunit.client.0.smithi118.stderr:HEAD is now at 1c349451176 Merge pull request #56077 from idryomov/wip-rbd-mirror-bash 2024-03-12T03:24:23.625 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-12T03:24:23.750 INFO:tasks.workunit.client.0.smithi118.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-12T03:24:23.752 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-12T03:24:23.752 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-12T03:24:23.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:23 smithi177 ceph-mon[37140]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:24:24.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:23 smithi118 ceph-mon[29160]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-12T03:24:24.712 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-12T03:24:24.805 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-12T03:24:24.848 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-12T03:24:24.849 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-12T03:24:24.849 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-12T03:24:24.890 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-12T03:24:24.893 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:24:24.893 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-12T03:24:24.927 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-12T03:24:24.928 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-12T03:24:24.928 DEBUG:teuthology.orchestra.run.smithi118:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-12T03:24:24.987 INFO:tasks.workunit.client.0.smithi118.stderr:+ mkdir -p fsstress 2024-03-12T03:24:24.988 INFO:tasks.workunit.client.0.smithi118.stderr:+ pushd fsstress 2024-03-12T03:24:24.988 INFO:tasks.workunit.client.0.smithi118.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-12T03:24:24.989 INFO:tasks.workunit.client.0.smithi118.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-12T03:24:25.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:24 smithi177 ceph-mon[37140]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:24:25.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:24 smithi118 ceph-mon[29160]: pgmap v233: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-12T03:24:27.948 INFO:tasks.workunit.client.1.smithi177.stderr:+ tar xzf ltp-full.tgz 2024-03-12T03:24:27.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:27 smithi177 ceph-mon[37140]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:24:28.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:27 smithi118 ceph-mon[29160]: pgmap v234: 65 pgs: 65 active+clean; 589 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-12T03:24:29.774 INFO:tasks.workunit.client.0.smithi118.stderr:+ tar xzf ltp-full.tgz 2024-03-12T03:24:29.849 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:29 smithi177 ceph-mon[37140]: pgmap v235: 65 pgs: 65 active+clean; 1.9 MiB data, 163 MiB used, 536 GiB / 536 GiB avail; 4.9 KiB/s rd, 114 KiB/s wr, 5 op/s 2024-03-12T03:24:30.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:29 smithi118 ceph-mon[29160]: pgmap v235: 65 pgs: 65 active+clean; 1.9 MiB data, 163 MiB used, 536 GiB / 536 GiB avail; 4.9 KiB/s rd, 114 KiB/s wr, 5 op/s 2024-03-12T03:24:31.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:31 smithi177 ceph-mon[37140]: pgmap v236: 65 pgs: 65 active+clean; 9.0 MiB data, 176 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 735 KiB/s wr, 24 op/s 2024-03-12T03:24:31.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:31 smithi177 ceph-mon[37140]: Upgrade: Updating mgr.smithi177.tmmsuz 2024-03-12T03:24:31.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:31 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:31.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:31 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi177.tmmsuz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:24:31.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:31 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:24:31.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:31 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:24:31.703 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:31 smithi177 ceph-mon[37140]: Deploying daemon mgr.smithi177.tmmsuz on smithi177 2024-03-12T03:24:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:31 smithi118 ceph-mon[29160]: pgmap v236: 65 pgs: 65 active+clean; 9.0 MiB data, 176 MiB used, 536 GiB / 536 GiB avail; 4.5 KiB/s rd, 735 KiB/s wr, 24 op/s 2024-03-12T03:24:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:31 smithi118 ceph-mon[29160]: Upgrade: Updating mgr.smithi177.tmmsuz 2024-03-12T03:24:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:31 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:31 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi177.tmmsuz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:24:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:31 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:24:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:31 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:24:32.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:31 smithi118 ceph-mon[29160]: Deploying daemon mgr.smithi177.tmmsuz on smithi177 2024-03-12T03:24:32.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/3317264006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0103e40 msgr2=0x7f7aa0106230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/3317264006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0103e40 0x7f7aa0106230 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7a94009910 tx=0x7f7a94009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/3317264006 shutdown_connections 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/3317264006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0103e40 0x7f7aa0106230 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/3317264006 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7aa0101510 0x7f7aa0103900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/3317264006 >> 172.21.15.118:0/3317264006 conn(0x7f7aa00faf60 msgr2=0x7f7aa00fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/3317264006 shutdown_connections 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.095+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/3317264006 wait complete. 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7aa5aa5700 1 Processor -- start 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7aa5aa5700 1 -- start start 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7aa5aa5700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0101510 0x7f7aa0196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7aa5aa5700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7aa0103e40 0x7f7aa0196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7aa5aa5700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa0196fa0 con 0x7f7aa0101510 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7aa5aa5700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa01970e0 con 0x7f7aa0103e40 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7a9f7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0101510 0x7f7aa0196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7a9f7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0101510 0x7f7aa0196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43202/0 (socket says 172.21.15.118:43202) 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7a9f7fe700 1 -- 172.21.15.118:0/1033244856 learned_addr learned my addr 172.21.15.118:0/1033244856 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.096+0000 7f7a9effd700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7aa0103e40 0x7f7aa0196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7a9f7fe700 1 -- 172.21.15.118:0/1033244856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7aa0103e40 msgr2=0x7f7aa0196980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7a9f7fe700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7aa0103e40 0x7f7aa0196980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7a9f7fe700 1 -- 172.21.15.118:0/1033244856 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a940095f0 con 0x7f7aa0101510 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7a9f7fe700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0101510 0x7f7aa0196440 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000d8d0 tx=0x7f7a9000dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7a9cff9700 1 -- 172.21.15.118:0/1033244856 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a900098a0 con 0x7f7aa0101510 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7a9cff9700 1 -- 172.21.15.118:0/1033244856 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a90010460 con 0x7f7aa0101510 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa019bb40 con 0x7f7aa0101510 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7a9cff9700 1 -- 172.21.15.118:0/1033244856 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a9000f5f0 con 0x7f7aa0101510 2024-03-12T03:24:32.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.097+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa00ff300 con 0x7f7aa0101510 2024-03-12T03:24:32.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.098+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa00623e0 con 0x7f7aa0101510 2024-03-12T03:24:32.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.099+0000 7f7a9cff9700 1 -- 172.21.15.118:0/1033244856 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f7a90009a00 con 0x7f7aa0101510 2024-03-12T03:24:32.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.099+0000 7f7a9cff9700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7a8806c880 0x7f7a8806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.099+0000 7f7a9cff9700 1 -- 172.21.15.118:0/1033244856 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f7a9008bba0 con 0x7f7aa0101510 2024-03-12T03:24:32.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.099+0000 7f7a9effd700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7a8806c880 0x7f7a8806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.100+0000 7f7a9effd700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7a8806c880 0x7f7a8806ed40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a94005090 tx=0x7f7a94005960 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:32.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.104+0000 7f7a9cff9700 1 -- 172.21.15.118:0/1033244856 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f7a90059be0 con 0x7f7aa0101510 2024-03-12T03:24:32.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.394+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f7aa0105d40 con 0x7f7a8806c880 2024-03-12T03:24:32.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.395+0000 7f7a9cff9700 1 -- 172.21.15.118:0/1033244856 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f7aa0105d40 con 0x7f7a8806c880 2024-03-12T03:24:32.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7a8806c880 msgr2=0x7f7a8806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7a8806c880 0x7f7a8806ed40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7a94005090 tx=0x7f7a94005960 comp rx=0 tx=0).stop 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0101510 msgr2=0x7f7aa0196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0101510 0x7f7aa0196440 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000d8d0 tx=0x7f7a9000dbe0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 shutdown_connections 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f7a8806c880 0x7f7a8806ed40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa0101510 0x7f7aa0196440 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 --2- 172.21.15.118:0/1033244856 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f7aa0103e40 0x7f7aa0196980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 >> 172.21.15.118:0/1033244856 conn(0x7f7aa00faf60 msgr2=0x7f7aa00fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 shutdown_connections 2024-03-12T03:24:32.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.398+0000 7f7aa5aa5700 1 -- 172.21.15.118:0/1033244856 wait complete. 2024-03-12T03:24:32.416 INFO:teuthology.orchestra.run.smithi118.stdout:true 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 -- 172.21.15.118:0/1443581053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa80ff980 msgr2=0x7fefa80ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 --2- 172.21.15.118:0/1443581053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa80ff980 0x7fefa80ffda0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fef98009910 tx=0x7fef98009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 -- 172.21.15.118:0/1443581053 shutdown_connections 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 --2- 172.21.15.118:0/1443581053 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefa8100b80 0x7fefa8100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 --2- 172.21.15.118:0/1443581053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa80ff980 0x7fefa80ffda0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 -- 172.21.15.118:0/1443581053 >> 172.21.15.118:0/1443581053 conn(0x7fefa80faf60 msgr2=0x7fefa80fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 -- 172.21.15.118:0/1443581053 shutdown_connections 2024-03-12T03:24:32.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.548+0000 7fefade92700 1 -- 172.21.15.118:0/1443581053 wait complete. 2024-03-12T03:24:32.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.549+0000 7fefade92700 1 Processor -- start 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.549+0000 7fefade92700 1 -- start start 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.549+0000 7fefade92700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefa80ff980 0x7fefa8196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.549+0000 7fefade92700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8100b80 0x7fefa81969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.549+0000 7fefade92700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8196ff0 con 0x7fefa8100b80 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.549+0000 7fefade92700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa8197130 con 0x7fefa80ff980 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.549+0000 7fefa6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8100b80 0x7fefa81969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8100b80 0x7fefa81969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43218/0 (socket says 172.21.15.118:43218) 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa6ffd700 1 -- 172.21.15.118:0/2385118001 learned_addr learned my addr 172.21.15.118:0/2385118001 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa77fe700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefa80ff980 0x7fefa8196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa6ffd700 1 -- 172.21.15.118:0/2385118001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefa80ff980 msgr2=0x7fefa8196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa6ffd700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefa80ff980 0x7fefa8196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa6ffd700 1 -- 172.21.15.118:0/2385118001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef980095f0 con 0x7fefa8100b80 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa77fe700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefa80ff980 0x7fefa8196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T03:24:32.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa6ffd700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8100b80 0x7fefa81969d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00e920 tx=0x7fef9c00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:32.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa4ff9700 1 -- 172.21.15.118:0/2385118001 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9c00c9d0 con 0x7fefa8100b80 2024-03-12T03:24:32.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefa819bb20 con 0x7fefa8100b80 2024-03-12T03:24:32.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa4ff9700 1 -- 172.21.15.118:0/2385118001 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef9c010430 con 0x7fefa8100b80 2024-03-12T03:24:32.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefa4ff9700 1 -- 172.21.15.118:0/2385118001 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9c018630 con 0x7fefa8100b80 2024-03-12T03:24:32.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.550+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa819c0f0 con 0x7fefa8100b80 2024-03-12T03:24:32.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.551+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa80623e0 con 0x7fefa8100b80 2024-03-12T03:24:32.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.552+0000 7fefa4ff9700 1 -- 172.21.15.118:0/2385118001 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7fef9c0105a0 con 0x7fefa8100b80 2024-03-12T03:24:32.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.553+0000 7fefa4ff9700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef9006c880 0x7fef9006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.553+0000 7fefa4ff9700 1 -- 172.21.15.118:0/2385118001 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7fef9c014070 con 0x7fefa8100b80 2024-03-12T03:24:32.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.553+0000 7fefa77fe700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef9006c880 0x7fef9006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.553+0000 7fefa77fe700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef9006c880 0x7fef9006ed40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fef9800b800 tx=0x7fef98009f70 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:32.561 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.558+0000 7fefa4ff9700 1 -- 172.21.15.118:0/2385118001 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7fef9c052960 con 0x7fefa8100b80 2024-03-12T03:24:32.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.839+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefa8068b10 con 0x7fef9006c880 2024-03-12T03:24:32.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.841+0000 7fefa4ff9700 1 -- 172.21.15.118:0/2385118001 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7fefa8068b10 con 0x7fef9006c880 2024-03-12T03:24:32.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef9006c880 msgr2=0x7fef9006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef9006c880 0x7fef9006ed40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fef9800b800 tx=0x7fef98009f70 comp rx=0 tx=0).stop 2024-03-12T03:24:32.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8100b80 msgr2=0x7fefa81969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8100b80 0x7fefa81969d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fef9c00e920 tx=0x7fef9c00ec30 comp rx=0 tx=0).stop 2024-03-12T03:24:32.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 shutdown_connections 2024-03-12T03:24:32.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7fef9006c880 0x7fef9006ed40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fefa80ff980 0x7fefa8196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 --2- 172.21.15.118:0/2385118001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefa8100b80 0x7fefa81969d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 >> 172.21.15.118:0/2385118001 conn(0x7fefa80faf60 msgr2=0x7fefa8068400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:32.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 shutdown_connections 2024-03-12T03:24:32.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.843+0000 7fefade92700 1 -- 172.21.15.118:0/2385118001 wait complete. 2024-03-12T03:24:32.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 -- 172.21.15.118:0/2491487944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ffcc0 msgr2=0x7f5454100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 --2- 172.21.15.118:0/2491487944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ffcc0 0x7f5454100140 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f5440009910 tx=0x7f5440009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 -- 172.21.15.118:0/2491487944 shutdown_connections 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 --2- 172.21.15.118:0/2491487944 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ffcc0 0x7f5454100140 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 --2- 172.21.15.118:0/2491487944 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54540ff360 0x7f54540ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 -- 172.21.15.118:0/2491487944 >> 172.21.15.118:0/2491487944 conn(0x7f54540faf60 msgr2=0x7f54540fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 -- 172.21.15.118:0/2491487944 shutdown_connections 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.964+0000 7f545b210700 1 -- 172.21.15.118:0/2491487944 wait complete. 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f545b210700 1 Processor -- start 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f545b210700 1 -- start start 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f545b210700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ff360 0x7f5454198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f545b210700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54540ffcc0 0x7f5454198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f545b210700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54541991f0 con 0x7f54540ff360 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f545b210700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5454199330 con 0x7f54540ffcc0 2024-03-12T03:24:32.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f5458fac700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ff360 0x7f5454198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f5458fac700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ff360 0x7f5454198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43224/0 (socket says 172.21.15.118:43224) 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.965+0000 7f5458fac700 1 -- 172.21.15.118:0/2346526167 learned_addr learned my addr 172.21.15.118:0/2346526167 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5453fff700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54540ffcc0 0x7f5454198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5458fac700 1 -- 172.21.15.118:0/2346526167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54540ffcc0 msgr2=0x7f5454198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5458fac700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54540ffcc0 0x7f5454198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5458fac700 1 -- 172.21.15.118:0/2346526167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54400095f0 con 0x7f54540ff360 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5453fff700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54540ffcc0 0x7f5454198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5458fac700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ff360 0x7f5454198690 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f544800e920 tx=0x7f544800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5451ffb700 1 -- 172.21.15.118:0/2346526167 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5448009790 con 0x7f54540ff360 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5451ffb700 1 -- 172.21.15.118:0/2346526167 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f544800cba0 con 0x7f54540ff360 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f5451ffb700 1 -- 172.21.15.118:0/2346526167 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5448018650 con 0x7f54540ff360 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.966+0000 7f545b210700 1 -- 172.21.15.118:0/2346526167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545419dd40 con 0x7f54540ff360 2024-03-12T03:24:32.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.967+0000 7f545b210700 1 -- 172.21.15.118:0/2346526167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545419e2b0 con 0x7f54540ff360 2024-03-12T03:24:32.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.968+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54340052f0 con 0x7f54540ff360 2024-03-12T03:24:32.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.969+0000 7f5451ffb700 1 -- 172.21.15.118:0/2346526167 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f544800cd10 con 0x7f54540ff360 2024-03-12T03:24:32.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.969+0000 7f5451ffb700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f544406c880 0x7f544406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:32.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.969+0000 7f5451ffb700 1 -- 172.21.15.118:0/2346526167 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f5448014070 con 0x7f54540ff360 2024-03-12T03:24:32.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.969+0000 7f5453fff700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f544406c880 0x7f544406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:32.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.970+0000 7f5453fff700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f544406c880 0x7f544406ed40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5440000c00 tx=0x7f5440005c00 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:32.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:32.974+0000 7f5451ffb700 1 -- 172.21.15.118:0/2346526167 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f5448052960 con 0x7f54540ff360 2024-03-12T03:24:33.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.223+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f5434000bc0 con 0x7f544406c880 2024-03-12T03:24:33.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.239+0000 7f5451ffb700 1 -- 172.21.15.118:0/2346526167 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3840 (secure 0 0 0) 0x7f5434000bc0 con 0x7f544406c880 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:alertmanager.smithi118 smithi118 *:9093,9094 running (5m) 85s ago 7m 24.9M - 0.25.0 c8568f914cd2 ab6c8c340db8 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-exporter.smithi118 smithi118 running (7m) 85s ago 7m 7507k - 18.2.2-751-g62a11b3c fa4cc2e02803 dab7493ee414 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-exporter.smithi177 smithi177 running (6m) 88s ago 6m 7453k - 18.2.2-751-g62a11b3c fa4cc2e02803 56905fbf68ab 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:crash.smithi118 smithi118 running (7m) 85s ago 7m 6660k - 18.2.2-751-g62a11b3c fa4cc2e02803 848a90530bcc 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:crash.smithi177 smithi177 running (6m) 88s ago 6m 6660k - 18.2.2-751-g62a11b3c fa4cc2e02803 73a879109cfd 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:grafana.smithi118 smithi118 *:3000 running (5m) 85s ago 7m 83.8M - 9.4.7 954c08fa6188 f56cb9df4974 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi118.oysrod smithi118 running (104s) 85s ago 104s 18.1M - 18.2.2-751-g62a11b3c fa4cc2e02803 0579203a308a 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi118.wouiai smithi118 running (112s) 85s ago 111s 13.0M - 18.2.2-751-g62a11b3c fa4cc2e02803 47ff6106211e 2024-03-12T03:24:33.242 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi177.azybll smithi177 running (102s) 88s ago 101s 16.6M - 18.2.2-751-g62a11b3c fa4cc2e02803 237998f5045b 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:mds.cephfs.smithi177.kbzhid smithi177 running (108s) 88s ago 108s 16.0M - 18.2.2-751-g62a11b3c fa4cc2e02803 71614295c01b 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.smithi118.zzzuvl smithi118 *:9283,8765,8443 running (9m) 85s ago 9m 497M - 18.2.2-751-g62a11b3c fa4cc2e02803 b31fb006922b 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.smithi177.tmmsuz smithi177 *:8443,9283,8765 running (6m) 88s ago 6m 453M - 18.2.2-751-g62a11b3c fa4cc2e02803 9f6d8c95b283 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:mon.smithi118 smithi118 running (9m) 85s ago 9m 45.5M 2048M 18.2.2-751-g62a11b3c fa4cc2e02803 a26ddd75d886 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:mon.smithi177 smithi177 running (6m) 88s ago 6m 36.9M 2048M 18.2.2-751-g62a11b3c fa4cc2e02803 0d8acd4f44b3 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:node-exporter.smithi118 smithi118 *:9100 running (7m) 85s ago 7m 16.3M - 1.5.0 0da6a335fe13 4e2acb3f4a85 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:node-exporter.smithi177 smithi177 *:9100 running (6m) 88s ago 6m 16.8M - 1.5.0 0da6a335fe13 bde9c5ddfe9c 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:osd.0 smithi118 running (5m) 85s ago 5m 50.4M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 9436f155422d 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:osd.1 smithi118 running (5m) 85s ago 5m 48.2M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 38fd187fe41b 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:osd.2 smithi118 running (4m) 85s ago 4m 46.0M 3946M 18.2.2-751-g62a11b3c fa4cc2e02803 a5856da80958 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:osd.3 smithi177 running (4m) 88s ago 4m 47.6M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 1a594e7be73e 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:osd.4 smithi177 running (3m) 88s ago 3m 47.0M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 a4b1d7721078 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:osd.5 smithi177 running (3m) 88s ago 3m 43.5M 4970M 18.2.2-751-g62a11b3c fa4cc2e02803 fd4d4398ebeb 2024-03-12T03:24:33.243 INFO:teuthology.orchestra.run.smithi118.stdout:prometheus.smithi118 smithi118 *:9095 running (5m) 85s ago 6m 53.6M - 2.43.0 a07b618ecd1d 367c172183f1 2024-03-12T03:24:33.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f544406c880 msgr2=0x7f544406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f544406c880 0x7f544406ed40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5440000c00 tx=0x7f5440005c00 comp rx=0 tx=0).stop 2024-03-12T03:24:33.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ff360 msgr2=0x7f5454198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ff360 0x7f5454198690 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f544800e920 tx=0x7f544800ece0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 shutdown_connections 2024-03-12T03:24:33.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f544406c880 0x7f544406ed40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54540ff360 0x7f5454198690 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 --2- 172.21.15.118:0/2346526167 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f54540ffcc0 0x7f5454198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 >> 172.21.15.118:0/2346526167 conn(0x7f54540faf60 msgr2=0x7f5454107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:33.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 shutdown_connections 2024-03-12T03:24:33.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.243+0000 7f543f7fe700 1 -- 172.21.15.118:0/2346526167 wait complete. 2024-03-12T03:24:33.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 -- 172.21.15.118:0/3305632460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 msgr2=0x7f18a8103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/3305632460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 0x7f18a8103960 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1898009910 tx=0x7f1898009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:33.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 -- 172.21.15.118:0/3305632460 shutdown_connections 2024-03-12T03:24:33.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/3305632460 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a8103ea0 0x7f18a8106290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/3305632460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 0x7f18a8103960 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 -- 172.21.15.118:0/3305632460 >> 172.21.15.118:0/3305632460 conn(0x7f18a80faf60 msgr2=0x7f18a80fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:33.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 -- 172.21.15.118:0/3305632460 shutdown_connections 2024-03-12T03:24:33.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.392+0000 7f18ad4bd700 1 -- 172.21.15.118:0/3305632460 wait complete. 2024-03-12T03:24:33.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.393+0000 7f18ad4bd700 1 Processor -- start 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.393+0000 7f18ad4bd700 1 -- start start 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18ad4bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 0x7f18a8196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18ad4bd700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a8103ea0 0x7f18a8196980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18ad4bd700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a8196fa0 con 0x7f18a8101570 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18ad4bd700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a81970e0 con 0x7f18a8103ea0 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 0x7f18a8196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a6ffd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 0x7f18a8196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43242/0 (socket says 172.21.15.118:43242) 2024-03-12T03:24:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a6ffd700 1 -- 172.21.15.118:0/2044012979 learned_addr learned my addr 172.21.15.118:0/2044012979 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a67fc700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a8103ea0 0x7f18a8196980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a67fc700 1 -- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 msgr2=0x7f18a8196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a67fc700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 0x7f18a8196440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a67fc700 1 -- 172.21.15.118:0/2044012979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18980095f0 con 0x7f18a8103ea0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.394+0000 7f18a67fc700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a8103ea0 0x7f18a8196980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f189c007a60 tx=0x7f189c007d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.395+0000 7f188ffff700 1 -- 172.21.15.118:0/2044012979 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f189c014070 con 0x7f18a8103ea0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.395+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a80ff3c0 con 0x7f18a8103ea0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.395+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a80ff910 con 0x7f18a8103ea0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.395+0000 7f188ffff700 1 -- 172.21.15.118:0/2044012979 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f189c004830 con 0x7f18a8103ea0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.395+0000 7f188ffff700 1 -- 172.21.15.118:0/2044012979 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f189c005620 con 0x7f18a8103ea0 2024-03-12T03:24:33.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.396+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1894005320 con 0x7f18a8103ea0 2024-03-12T03:24:33.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.397+0000 7f188ffff700 1 -- 172.21.15.118:0/2044012979 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f189c018070 con 0x7f18a8103ea0 2024-03-12T03:24:33.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.397+0000 7f188ffff700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f189006c880 0x7f189006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:33.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.397+0000 7f188ffff700 1 -- 172.21.15.118:0/2044012979 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f189c08efd0 con 0x7f18a8103ea0 2024-03-12T03:24:33.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.397+0000 7f18a6ffd700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f189006c880 0x7f189006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:33.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.398+0000 7f18a6ffd700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f189006c880 0x7f189006ed40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f18980051c0 tx=0x7f189800b540 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:33.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.403+0000 7f188ffff700 1 -- 172.21.15.118:0/2044012979 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f189c05d010 con 0x7f18a8103ea0 2024-03-12T03:24:33.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.725+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f1894006200 con 0x7f18a8103ea0 2024-03-12T03:24:33.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.726+0000 7f188ffff700 1 -- 172.21.15.118:0/2044012979 <== mon.1 v2:172.21.15.177:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+628 (secure 0 0 0) 0x7f189c05cba0 con 0x7f18a8103ea0 2024-03-12T03:24:33.728 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-12T03:24:33.728 INFO:teuthology.orchestra.run.smithi118.stdout: "mon": { 2024-03-12T03:24:33.728 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 2 2024-03-12T03:24:33.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "mgr": { 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 2 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "osd": { 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 6 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "mds": { 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 4 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "overall": { 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph version 18.2.2-751-g62a11b3c (62a11b3cb13eae2a420414a5cf4f907b6bc88089) reef (stable)": 14 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-12T03:24:33.729 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-12T03:24:33.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f189006c880 msgr2=0x7f189006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f189006c880 0x7f189006ed40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f18980051c0 tx=0x7f189800b540 comp rx=0 tx=0).stop 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a8103ea0 msgr2=0x7f18a8196980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a8103ea0 0x7f18a8196980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f189c007a60 tx=0x7f189c007d70 comp rx=0 tx=0).stop 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 shutdown_connections 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f189006c880 0x7f189006ed40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f18a8101570 0x7f18a8196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 --2- 172.21.15.118:0/2044012979 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f18a8103ea0 0x7f18a8196980 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 >> 172.21.15.118:0/2044012979 conn(0x7f18a80faf60 msgr2=0x7f18a80fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 shutdown_connections 2024-03-12T03:24:33.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.729+0000 7f18ad4bd700 1 -- 172.21.15.118:0/2044012979 wait complete. 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 -- 172.21.15.118:0/1611605731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0101cb0 msgr2=0x7f4eb0102110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/1611605731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0101cb0 0x7f4eb0102110 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009910 tx=0x7f4ea0009c20 comp rx=0 tx=0).stop 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 -- 172.21.15.118:0/1611605731 shutdown_connections 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/1611605731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0101cb0 0x7f4eb0102110 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/1611605731 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eb0100ab0 0x7f4eb0100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 -- 172.21.15.118:0/1611605731 >> 172.21.15.118:0/1611605731 conn(0x7f4eb00fc050 msgr2=0x7f4eb00fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 -- 172.21.15.118:0/1611605731 shutdown_connections 2024-03-12T03:24:33.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.854+0000 7f4eb7acb700 1 -- 172.21.15.118:0/1611605731 wait complete. 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.855+0000 7f4eb7acb700 1 Processor -- start 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.855+0000 7f4eb7acb700 1 -- start start 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb7acb700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0100ab0 0x7f4eb01986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb7acb700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eb0101cb0 0x7f4eb0198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb7acb700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb0199250 con 0x7f4eb0100ab0 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb7acb700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb0199390 con 0x7f4eb0101cb0 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5867700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0100ab0 0x7f4eb01986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5867700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0100ab0 0x7f4eb01986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43260/0 (socket says 172.21.15.118:43260) 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5867700 1 -- 172.21.15.118:0/2913708143 learned_addr learned my addr 172.21.15.118:0/2913708143 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:24:33.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5066700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eb0101cb0 0x7f4eb0198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5066700 1 -- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0100ab0 msgr2=0x7f4eb01986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5066700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0100ab0 0x7f4eb01986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5066700 1 -- 172.21.15.118:0/2913708143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea00095f0 con 0x7f4eb0101cb0 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5867700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0100ab0 0x7f4eb01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.856+0000 7f4eb5066700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eb0101cb0 0x7f4eb0198c30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0006010 tx=0x7f4ea0005f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.857+0000 7f4ea6ffd700 1 -- 172.21.15.118:0/2913708143 <== mon.1 v2:172.21.15.177:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ea001d070 con 0x7f4eb0101cb0 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.857+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb019dd40 con 0x7f4eb0101cb0 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.857+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb019e2b0 con 0x7f4eb0101cb0 2024-03-12T03:24:33.860 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.857+0000 7f4ea6ffd700 1 -- 172.21.15.118:0/2913708143 <== mon.1 v2:172.21.15.177:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ea00052a0 con 0x7f4eb0101cb0 2024-03-12T03:24:33.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.857+0000 7f4ea6ffd700 1 -- 172.21.15.118:0/2913708143 <== mon.1 v2:172.21.15.177:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ea0017400 con 0x7f4eb0101cb0 2024-03-12T03:24:33.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.858+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e94005320 con 0x7f4eb0101cb0 2024-03-12T03:24:33.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.859+0000 7f4ea6ffd700 1 -- 172.21.15.118:0/2913708143 <== mon.1 v2:172.21.15.177:3300/0 4 ==== mgrmap(e 19) v1 ==== 90526+0+0 (secure 0 0 0) 0x7f4ea0017630 con 0x7f4eb0101cb0 2024-03-12T03:24:33.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.859+0000 7f4ea6ffd700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4e9c06c880 0x7f4e9c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:24:33.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.859+0000 7f4ea6ffd700 1 -- 172.21.15.118:0/2913708143 <== mon.1 v2:172.21.15.177:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5554+0+0 (secure 0 0 0) 0x7f4ea008ce10 con 0x7f4eb0101cb0 2024-03-12T03:24:33.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.859+0000 7f4eb5867700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4e9c06c880 0x7f4e9c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:24:33.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.860+0000 7f4eb5867700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4e9c06c880 0x7f4e9c06ed40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009fd0 tx=0x7f4eac0091e0 comp rx=0 tx=0).ready entity=mgr.14221 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:24:33.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:33.865+0000 7f4ea6ffd700 1 -- 172.21.15.118:0/2913708143 <== mon.1 v2:172.21.15.177:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182595 (secure 0 0 0) 0x7f4ea005b090 con 0x7f4eb0101cb0 2024-03-12T03:24:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:33 smithi177 ceph-mon[37140]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:33 smithi177 ceph-mon[37140]: pgmap v237: 65 pgs: 65 active+clean; 19 MiB data, 211 MiB used, 536 GiB / 536 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 63 op/s 2024-03-12T03:24:33.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:33 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:34.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:33 smithi118 ceph-mon[29160]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:34.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:33 smithi118 ceph-mon[29160]: pgmap v237: 65 pgs: 65 active+clean; 19 MiB data, 211 MiB used, 536 GiB / 536 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 63 op/s 2024-03-12T03:24:34.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:33 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:34.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.128+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 --> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e94000bf0 con 0x7f4e9c06c880 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.129+0000 7f4ea6ffd700 1 -- 172.21.15.118:0/2913708143 <== mgr.14221 v2:172.21.15.118:6800/3827716556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f4e94000bf0 con 0x7f4e9c06c880 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379", 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout: "in_progress": true, 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout: "services_complete": [], 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout: "progress": "0/2 daemons upgraded", 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout: "message": "Currently upgrading mgr daemons", 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout: "is_paused": false 2024-03-12T03:24:34.132 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-12T03:24:34.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.131+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4e9c06c880 msgr2=0x7f4e9c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:34.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.131+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4e9c06c880 0x7f4e9c06ed40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009fd0 tx=0x7f4eac0091e0 comp rx=0 tx=0).stop 2024-03-12T03:24:34.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.131+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eb0101cb0 msgr2=0x7f4eb0198c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:24:34.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eb0101cb0 0x7f4eb0198c30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0006010 tx=0x7f4ea0005f00 comp rx=0 tx=0).stop 2024-03-12T03:24:34.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 shutdown_connections 2024-03-12T03:24:34.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:6800/3827716556,v1:172.21.15.118:6801/3827716556] conn(0x7f4e9c06c880 0x7f4e9c06ed40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:34.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4eb0100ab0 0x7f4eb01986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:34.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 --2- 172.21.15.118:0/2913708143 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7f4eb0101cb0 0x7f4eb0198c30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:24:34.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 >> 172.21.15.118:0/2913708143 conn(0x7f4eb00fc050 msgr2=0x7f4eb01032f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:24:34.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 shutdown_connections 2024-03-12T03:24:34.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:24:34.132+0000 7f4eb7acb700 1 -- 172.21.15.118:0/2913708143 wait complete. 2024-03-12T03:24:34.924 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:34 smithi177 ceph-mon[37140]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:34.924 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:34 smithi177 ceph-mon[37140]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:34.924 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:34 smithi177 ceph-mon[37140]: from='client.? 172.21.15.118:0/2044012979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:24:35.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:34 smithi118 ceph-mon[29160]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:35.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:34 smithi118 ceph-mon[29160]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:35.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:34 smithi118 ceph-mon[29160]: from='client.? 172.21.15.118:0/2044012979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:24:35.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:35 smithi177 ceph-mon[37140]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:35.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:35 smithi177 ceph-mon[37140]: pgmap v238: 65 pgs: 65 active+clean; 34 MiB data, 275 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.9 MiB/s wr, 110 op/s 2024-03-12T03:24:36.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:35 smithi118 ceph-mon[29160]: from='client.24375 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:24:36.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:35 smithi118 ceph-mon[29160]: pgmap v238: 65 pgs: 65 active+clean; 34 MiB data, 275 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 2.9 MiB/s wr, 110 op/s 2024-03-12T03:24:37.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:37 smithi177 ceph-mon[37140]: pgmap v239: 65 pgs: 65 active+clean; 38 MiB data, 291 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 121 op/s 2024-03-12T03:24:38.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:37 smithi118 ceph-mon[29160]: pgmap v239: 65 pgs: 65 active+clean; 38 MiB data, 291 MiB used, 536 GiB / 536 GiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 121 op/s 2024-03-12T03:24:38.978 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:38.978 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:38.978 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:38 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:39.118 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:39.118 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:39.119 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:38 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:40.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:39 smithi118 ceph-mon[29160]: pgmap v240: 65 pgs: 65 active+clean; 48 MiB data, 311 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 128 op/s 2024-03-12T03:24:40.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:39 smithi177 ceph-mon[37140]: pgmap v240: 65 pgs: 65 active+clean; 48 MiB data, 311 MiB used, 536 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.0 MiB/s wr, 128 op/s 2024-03-12T03:24:41.937 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:41 smithi177 ceph-mon[37140]: pgmap v241: 65 pgs: 65 active+clean; 55 MiB data, 316 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 185 op/s 2024-03-12T03:24:42.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:41 smithi118 ceph-mon[29160]: pgmap v241: 65 pgs: 65 active+clean; 55 MiB data, 316 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 4.6 MiB/s wr, 185 op/s 2024-03-12T03:24:43.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:43 smithi118 ceph-mon[29160]: pgmap v242: 65 pgs: 65 active+clean; 65 MiB data, 378 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 379 op/s 2024-03-12T03:24:44.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:43 smithi177 ceph-mon[37140]: pgmap v242: 65 pgs: 65 active+clean; 65 MiB data, 378 MiB used, 536 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.0 MiB/s wr, 379 op/s 2024-03-12T03:24:45.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:44 smithi177 ceph-mon[37140]: pgmap v243: 65 pgs: 65 active+clean; 100 MiB data, 562 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 7.2 MiB/s wr, 806 op/s 2024-03-12T03:24:45.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:44 smithi177 ceph-mon[37140]: Standby manager daemon smithi177.tmmsuz restarted 2024-03-12T03:24:45.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:44 smithi177 ceph-mon[37140]: Standby manager daemon smithi177.tmmsuz started 2024-03-12T03:24:45.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:44 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/crt"}]: dispatch 2024-03-12T03:24:45.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:44 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:24:45.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:44 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/key"}]: dispatch 2024-03-12T03:24:45.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:44 smithi177 ceph-mon[37140]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:24:45.277 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:44 smithi118 ceph-mon[29160]: pgmap v243: 65 pgs: 65 active+clean; 100 MiB data, 562 MiB used, 536 GiB / 536 GiB avail; 3.0 MiB/s rd, 7.2 MiB/s wr, 806 op/s 2024-03-12T03:24:45.277 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:44 smithi118 ceph-mon[29160]: Standby manager daemon smithi177.tmmsuz restarted 2024-03-12T03:24:45.277 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:44 smithi118 ceph-mon[29160]: Standby manager daemon smithi177.tmmsuz started 2024-03-12T03:24:45.277 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:44 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/crt"}]: dispatch 2024-03-12T03:24:45.278 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:44 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:24:45.278 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:44 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi177.tmmsuz/key"}]: dispatch 2024-03-12T03:24:45.278 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:44 smithi118 ceph-mon[29160]: from='mgr.? 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:24:46.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:45 smithi177 ceph-mon[37140]: mgrmap e20: smithi118.zzzuvl(active, since 6m), standbys: smithi177.tmmsuz 2024-03-12T03:24:46.310 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:45 smithi118 ceph-mon[29160]: mgrmap e20: smithi118.zzzuvl(active, since 6m), standbys: smithi177.tmmsuz 2024-03-12T03:24:47.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:46 smithi177 ceph-mon[37140]: pgmap v244: 65 pgs: 65 active+clean; 109 MiB data, 581 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 865 op/s 2024-03-12T03:24:47.310 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:46 smithi118 ceph-mon[29160]: pgmap v244: 65 pgs: 65 active+clean; 109 MiB data, 581 MiB used, 536 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.7 MiB/s wr, 865 op/s 2024-03-12T03:24:49.202 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:49.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:49.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:49.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:48 smithi177 ceph-mon[37140]: pgmap v245: 65 pgs: 65 active+clean; 122 MiB data, 637 MiB used, 536 GiB / 536 GiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 935 op/s 2024-03-12T03:24:49.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:49.203 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:48 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:48 smithi118 ceph-mon[29160]: pgmap v245: 65 pgs: 65 active+clean; 122 MiB data, 637 MiB used, 536 GiB / 536 GiB avail; 2.9 MiB/s rd, 7.5 MiB/s wr, 935 op/s 2024-03-12T03:24:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:48 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:51.798 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:51 smithi177 ceph-mon[37140]: pgmap v246: 65 pgs: 65 active+clean; 122 MiB data, 653 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 928 op/s 2024-03-12T03:24:52.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:51 smithi118 ceph-mon[29160]: pgmap v246: 65 pgs: 65 active+clean; 122 MiB data, 653 MiB used, 536 GiB / 536 GiB avail; 2.2 MiB/s rd, 6.7 MiB/s wr, 928 op/s 2024-03-12T03:24:53.411 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:53.411 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr fail", "who": "smithi118.zzzuvl"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: osdmap e48: 6 total, 6 up, 6 in 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "mgr fail", "who": "smithi118.zzzuvl"}]': finished 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: mgrmap e21: smithi177.tmmsuz(active, starting, since 0.00976431s) 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.oysrod"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.wouiai"}]: dispatch 2024-03-12T03:24:53.412 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.kbzhid"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.azybll"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mgr metadata", "who": "smithi177.tmmsuz", "id": "smithi177.tmmsuz"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: Manager daemon smithi177.tmmsuz is now available 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:53.413 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:53.414 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:24:53.414 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 ' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:24:53.414 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/trash_purge_schedule"}]: dispatch 2024-03-12T03:24:53.414 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:53 smithi118 ceph-mon[29160]: from='mgr.24379 ' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/trash_purge_schedule"}]: dispatch 2024-03-12T03:24:53.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:53.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:53.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:24:53.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:24:53.453 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd=[{"prefix": "mgr fail", "who": "smithi118.zzzuvl"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: osdmap e48: 6 total, 6 up, 6 in 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.14221 172.21.15.118:0/3216366011' entity='mgr.smithi118.zzzuvl' cmd='[{"prefix": "mgr fail", "who": "smithi118.zzzuvl"}]': finished 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: mgrmap e21: smithi177.tmmsuz(active, starting, since 0.00976431s) 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mon metadata", "id": "smithi118"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mon metadata", "id": "smithi177"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.oysrod"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi118.wouiai"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.kbzhid"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi177.azybll"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mgr metadata", "who": "smithi177.tmmsuz", "id": "smithi177.tmmsuz"}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:24:53.454 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: Manager daemon smithi177.tmmsuz is now available 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 ' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 172.21.15.177:0/2092759792' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/trash_purge_schedule"}]: dispatch 2024-03-12T03:24:53.455 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:53 smithi177 ceph-mon[37140]: from='mgr.24379 ' entity='mgr.smithi177.tmmsuz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi177.tmmsuz/trash_purge_schedule"}]: dispatch 2024-03-12T03:24:54.901 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:54 smithi118 ceph-mon[29160]: Deploying cephadm binary to smithi177 2024-03-12T03:24:54.901 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:54 smithi118 ceph-mon[29160]: Deploying cephadm binary to smithi118 2024-03-12T03:24:54.901 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:54 smithi118 ceph-mon[29160]: mgrmap e22: smithi177.tmmsuz(active, since 1.00883s) 2024-03-12T03:24:54.901 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:54 smithi118 ceph-mon[29160]: pgmap v3: 65 pgs: 65 active+clean; 138 MiB data, 704 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:24:54.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:54 smithi177 ceph-mon[37140]: Deploying cephadm binary to smithi177 2024-03-12T03:24:54.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:54 smithi177 ceph-mon[37140]: Deploying cephadm binary to smithi118 2024-03-12T03:24:54.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:54 smithi177 ceph-mon[37140]: mgrmap e22: smithi177.tmmsuz(active, since 1.00883s) 2024-03-12T03:24:54.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:54 smithi177 ceph-mon[37140]: pgmap v3: 65 pgs: 65 active+clean; 138 MiB data, 704 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:24:55.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:55 smithi118 ceph-mon[29160]: [12/Mar/2024:03:24:54] ENGINE Bus STARTING 2024-03-12T03:24:55.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:55 smithi118 ceph-mon[29160]: [12/Mar/2024:03:24:54] ENGINE Serving on https://172.21.15.177:7150 2024-03-12T03:24:55.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:55 smithi118 ceph-mon[29160]: [12/Mar/2024:03:24:54] ENGINE Serving on http://172.21.15.177:8765 2024-03-12T03:24:55.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:55 smithi118 ceph-mon[29160]: [12/Mar/2024:03:24:54] ENGINE Bus STARTED 2024-03-12T03:24:55.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:55 smithi118 ceph-mon[29160]: pgmap v4: 65 pgs: 65 active+clean; 138 MiB data, 704 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:24:55.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:55 smithi177 ceph-mon[37140]: [12/Mar/2024:03:24:54] ENGINE Bus STARTING 2024-03-12T03:24:55.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:55 smithi177 ceph-mon[37140]: [12/Mar/2024:03:24:54] ENGINE Serving on https://172.21.15.177:7150 2024-03-12T03:24:55.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:55 smithi177 ceph-mon[37140]: [12/Mar/2024:03:24:54] ENGINE Serving on http://172.21.15.177:8765 2024-03-12T03:24:55.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:55 smithi177 ceph-mon[37140]: [12/Mar/2024:03:24:54] ENGINE Bus STARTED 2024-03-12T03:24:55.953 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:55 smithi177 ceph-mon[37140]: pgmap v4: 65 pgs: 65 active+clean; 138 MiB data, 704 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:24:56.540 INFO:tasks.workunit.client.1.smithi177.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-12T03:24:56.552 INFO:tasks.workunit.client.1.smithi177.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-12T03:24:56.552 INFO:tasks.workunit.client.1.smithi177.stderr:+ make 2024-03-12T03:24:56.899 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:24:56 smithi118 ceph-mon[29160]: mgrmap e23: smithi177.tmmsuz(active, since 3s) 2024-03-12T03:24:56.952 INFO:journalctl@ceph.mon.smithi177.smithi177.stdout:Mar 12 03:24:56 smithi177 ceph-mon[37140]: mgrmap e23: smithi177.tmmsuz(active, since 3s) 2024-03-12T03:24:57.129 INFO:tasks.workunit.client.1.smithi177.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-12T03:25:04.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 -- 172.21.15.118:0/3760004835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb800100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-12T03:25:04.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 --2- 172.21.15.118:0/3760004835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ffcc0 0x7fb800100140 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb7f4009910 tx=0x7fb7f4009c20 comp rx=0 tx=0).stop 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 -- 172.21.15.118:0/3760004835 shutdown_connections 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 --2- 172.21.15.118:0/3760004835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ffcc0 0x7fb800100140 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 --2- 172.21.15.118:0/3760004835 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ff360 0x7fb8000ff780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 -- 172.21.15.118:0/3760004835 >> 172.21.15.118:0/3760004835 conn(0x7fb8000faf60 msgr2=0x7fb8000fd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 -- 172.21.15.118:0/3760004835 shutdown_connections 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 -- 172.21.15.118:0/3760004835 wait complete. 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.241+0000 7fb806233700 1 Processor -- start 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb806233700 1 -- start start 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb806233700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb800198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb806233700 1 --2- >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb800198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb806233700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800199380 con 0x7fb8000ff360 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb806233700 1 -- --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8001994c0 con 0x7fb8000ffcc0 2024-03-12T03:25:04.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7ff7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb800198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7ff7fe700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb800198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:54950/0 (socket says 172.21.15.118:54950) 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 learned_addr learned my addr 172.21.15.118:0/2308702459 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb800198d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb800198d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f40095f0 con 0x7fb8000ff360 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb800198820 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b4d0 tx=0x7fb7f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0011800 con 0x7fb8000ff360 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.242+0000 7fb806233700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80019ded0 con 0x7fb8000ff360 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.243+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0011e20 con 0x7fb8000ff360 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.243+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000f2e0 con 0x7fb8000ff360 2024-03-12T03:25:04.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.243+0000 7fb806233700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80019e440 con 0x7fb8000ff360 2024-03-12T03:25:04.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.244+0000 7fb806233700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8001928a0 con 0x7fb8000ff360 2024-03-12T03:25:04.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.244+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0011960 con 0x7fb8000ff360 2024-03-12T03:25:04.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.244+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:04.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.244+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5812+0+0 (secure 0 0 0) 0x7fb7f004cde0 con 0x7fb8000ff360 2024-03-12T03:25:04.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.249+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184148 (secure 0 0 0) 0x7fb7f00172c0 con 0x7fb8000ff360 2024-03-12T03:25:04.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:04.466+0000 7fb806233700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8000611f0 con 0x7fb7e8039230 2024-03-12T03:25:14.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:14.254+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.177:6828/588231664, fault. 2024-03-12T03:25:14.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:14.254+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-12T03:25:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:18.467+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:18.467+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:18.467+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:18.405+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:18.405+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:18.405+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:18.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb800198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb800198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb800198820 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b4d0 tx=0x7fb7f000b890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb800198820 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b4d0 tx=0x7fb7f000b890 comp rx=0 tx=0).stop 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb800198820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb800198820 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e80430b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8046430 con 0x7fb7e8040cc0 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8046570 con 0x7fb8000ffcc0 2024-03-12T03:25:18.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e80430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:18.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:18.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:18.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8046760 con 0x7fb7e8040cc0 2024-03-12T03:25:18.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e80430b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b220 tx=0x7fb7f0009210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:18.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f00172c0 con 0x7fb7e8040cc0 2024-03-12T03:25:18.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000fdf0 con 0x7fb7e8040cc0 2024-03-12T03:25:18.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:18.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000da10 con 0x7fb7e8040cc0 2024-03-12T03:25:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:19.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:19.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:19.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:19.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:19.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:19.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:20.520 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:20.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:20.521 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:20.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:20.521 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:20.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:20.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:20.398+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:20.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:20.398+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:20.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:20.398+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:20.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:20.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:20.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:21.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:21.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:21.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:21.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:21.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:21.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:21.447+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:21.447+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:21.447+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:21.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:21.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:21.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:22.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:22.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:22.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:22.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:22.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:22.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:22.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:22.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:22.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:22.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:22.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:22.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:23.502 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:23.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:23.502 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:23.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:23.502 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:23.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:23.503 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:23.465+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:23.503 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:23.465+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:23.503 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:23.465+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:23.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:23.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:23.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e80430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e80430b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e80430b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b220 tx=0x7fb7f0009210 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e80430b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b220 tx=0x7fb7f0009210 comp rx=0 tx=0).stop 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e80430b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e80430b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8048f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8049840 con 0x7fb8000ff360 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8049980 con 0x7fb8000ffcc0 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.924+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8048f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:23.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.925+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:23.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.925+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:23.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.925+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8049ac0 con 0x7fb8000ff360 2024-03-12T03:25:23.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.925+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8048f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009d40 tx=0x7fb7f004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:23.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f800bd30 con 0x7fb8000ff360 2024-03-12T03:25:23.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009e90 con 0x7fb8000ff360 2024-03-12T03:25:23.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:23.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800d4f0 con 0x7fb8000ff360 2024-03-12T03:25:24.462 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:24.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:24.463 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:24.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:24.463 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:24.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:24.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:24.465+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.177:6828/588231664, fault. 2024-03-12T03:25:24.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:24.465+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-12T03:25:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:24.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:24.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:24.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:24.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:24.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:24.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:25.473 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:25.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:25.473 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:25.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:25.473 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:25.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:25.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:25.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:25.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:25.471+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:25.471+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:25.471+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:26.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:26.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:26.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:26.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:26.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:26.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:26.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:26.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:26.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:26.568+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:26.568+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:26.568+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:27.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:27.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:27.551 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:27.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:27.551 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:27.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:27.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:27.406+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:27.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:27.406+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:27.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:27.406+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:27.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:27.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:27.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:28.204+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:28.204+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:28.204+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:28.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:28.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:28.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:28.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:28.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:28.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:28.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.924+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8048f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:25:28.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.924+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8048f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:28.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.924+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8048f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009d40 tx=0x7fb7f004c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:28.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.924+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8048f60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009d40 tx=0x7fb7f004c000 comp rx=0 tx=0).stop 2024-03-12T03:25:28.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.924+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8048f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:28.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.924+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8048f60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:28.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:28.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:28.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804a480 con 0x7fb7e8040cc0 2024-03-12T03:25:28.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804a5c0 con 0x7fb8000ffcc0 2024-03-12T03:25:28.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:28.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:28.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:28.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804a700 con 0x7fb7e8040cc0 2024-03-12T03:25:28.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003a0e0 tx=0x7fb7f80094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:28.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0024ea0 con 0x7fb7e8040cc0 2024-03-12T03:25:28.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0045430 con 0x7fb7e8040cc0 2024-03-12T03:25:28.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:28.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f003a210 con 0x7fb7e8040cc0 2024-03-12T03:25:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:29.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:29.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:29.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:29.199+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:29.199+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:29.199+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:29.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:29.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:29.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:30.247+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:30.247+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:30.247+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:30.407+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:30.407+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:30.407+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:30.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:30.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:30.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:31.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:31.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:31.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:31.365+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:31.365+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:31.365+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:31.639+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:31.639+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:31.639+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:32.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:32.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:32.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:32.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:32.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:32.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:32.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:32.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:32.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:33.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:33.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:33.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:33.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:33.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:33.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:33.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:33.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:33.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:33.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:33.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:33.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:33.634+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:33.634+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:33.634+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:33.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-12T03:25:33.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:33.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003a0e0 tx=0x7fb7f80094a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:33.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003a0e0 tx=0x7fb7f80094a0 comp rx=0 tx=0).stop 2024-03-12T03:25:33.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804af30 con 0x7fb8000ff360 2024-03-12T03:25:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b070 con 0x7fb8000ffcc0 2024-03-12T03:25:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e804a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:33.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:33.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:33.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804b1b0 con 0x7fb8000ff360 2024-03-12T03:25:33.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.927+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e804a890 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0011040 tx=0x7fb7f0011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:33.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f00178c0 con 0x7fb8000ff360 2024-03-12T03:25:33.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0017a20 con 0x7fb8000ff360 2024-03-12T03:25:33.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:33.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0012c20 con 0x7fb8000ff360 2024-03-12T03:25:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:34.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:34.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:34.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:34.239+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:34.239+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:34.239+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:34.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:34.876+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.177:6828/588231664, fault. 2024-03-12T03:25:34.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:34.876+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-12T03:25:34.878 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:34.612+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:34.878 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:34.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:34.879 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:34.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:35.317+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:35.317+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:35.317+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:35.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:35.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:35.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:35.644+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:35.644+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:35.644+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:36.280+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:36.280+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:36.280+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:36.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:36.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:36.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:36.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:36.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:36.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:37.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:37.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:37.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:37.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:37.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:37.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:37.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:37.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:37.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:38.543 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:38.256+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:38.543 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:38.256+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:38.543 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:38.256+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:38.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:38.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:38.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:38.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:38.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:38.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:38.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:38.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:38.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:38.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e804a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-12T03:25:38.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e804a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:38.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e804a890 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0011040 tx=0x7fb7f0011f90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:38.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e804a890 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0011040 tx=0x7fb7f0011f90 comp rx=0 tx=0).stop 2024-03-12T03:25:38.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:38.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e804a890 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:38.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:38.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:38.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b3c0 con 0x7fb7e8040cc0 2024-03-12T03:25:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b500 con 0x7fb8000ffcc0 2024-03-12T03:25:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e8045a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804b640 con 0x7fb7e8040cc0 2024-03-12T03:25:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0024820 tx=0x7fb7f0047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001bc00 con 0x7fb7e8040cc0 2024-03-12T03:25:38.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003ab00 con 0x7fb7e8040cc0 2024-03-12T03:25:38.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:38.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0044b90 con 0x7fb7e8040cc0 2024-03-12T03:25:39.511 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:39.221+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:39.511 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:39.221+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:39.511 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:39.221+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:39.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:39.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:39.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:39.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:39.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:39.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:39.508+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:39.508+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:39.508+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:40.235+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:40.235+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:40.235+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:40.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:40.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:40.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:40.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:40.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:40.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:40.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:40.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:40.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:41.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:41.220+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:41.487 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:41.220+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:41.487 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:41.220+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:41.487 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:41.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:41.487 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:41.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:41.487 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:41.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:41.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:41.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:41.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:42.248+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:42.248+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:42.248+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:42.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:42.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:42.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:42.372+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:42.372+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:42.372+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:43.515 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:43.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:43.515 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:43.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:43.515 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:43.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:43.515 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:43.387+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:43.515 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:43.387+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:43.515 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:43.387+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:43.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:43.406+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:43.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:43.406+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:43.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:43.406+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:43.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0024820 tx=0x7fb7f0047040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0024820 tx=0x7fb7f0047040 comp rx=0 tx=0).stop 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b600 con 0x7fb8000ff360 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.925+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b740 con 0x7fb8000ffcc0 2024-03-12T03:25:43.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.926+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:43.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.926+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:43.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.926+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:43.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.926+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804b880 con 0x7fb8000ff360 2024-03-12T03:25:43.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8040a70 tx=0x7fb7f0038040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:43.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009920 con 0x7fb8000ff360 2024-03-12T03:25:43.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800be90 con 0x7fb8000ff360 2024-03-12T03:25:43.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:43.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c460 con 0x7fb8000ff360 2024-03-12T03:25:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:44.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:44.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:44.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:44.247+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:44.247+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:44.247+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:44.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:44.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:44.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:44.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:44.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:44.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:45.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:45.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:45.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:45.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:45.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:45.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:45.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:45.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:45.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:45.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:45.678+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.177:6828/588231664, fault. 2024-03-12T03:25:45.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:45.678+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-12T03:25:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:46.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:46.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:46.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:46.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:46.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:46.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:46.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:46.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:46.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:46.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:46.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:46.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:47.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:47.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:47.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:47.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:47.333+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:47.333+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:47.333+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:47.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:47.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:47.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:47.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:47.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:47.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:48.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:48.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:48.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:48.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:48.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:48.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:48.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:48.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:48.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:48.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:25:48.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:48.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8040a70 tx=0x7fb7f0038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:48.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8040a70 tx=0x7fb7f0038040 comp rx=0 tx=0).stop 2024-03-12T03:25:48.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804bc20 con 0x7fb7e8040cc0 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804bd60 con 0x7fb8000ffcc0 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.926+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804bea0 con 0x7fb7e8040cc0 2024-03-12T03:25:48.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804c570 tx=0x7fb7f800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:48.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8024c60 con 0x7fb7e8040cc0 2024-03-12T03:25:48.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8024dc0 con 0x7fb7e8040cc0 2024-03-12T03:25:48.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:48.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8003ac0 con 0x7fb7e8040cc0 2024-03-12T03:25:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:49.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:49.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:49.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:49.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:49.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:49.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:49.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:49.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:49.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:50.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:50.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:50.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:50.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:50.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:50.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:50.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:50.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:50.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:51.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:51.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:51.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:51.264+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:51.264+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:51.264+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:51.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:51.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:51.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:52.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:52.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:52.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:52.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:52.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:52.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:52.263+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:52.263+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:52.263+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:53.515 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:53.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:53.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:53.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:53.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:53.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:53.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:53.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:53.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:53.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:53.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804c570 tx=0x7fb7f800b8d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804c570 tx=0x7fb7f800b8d0 comp rx=0 tx=0).stop 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:53.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:53.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c2f0 con 0x7fb8000ff360 2024-03-12T03:25:53.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c430 con 0x7fb8000ffcc0 2024-03-12T03:25:53.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:53.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.927+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:53.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.927+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:53.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.927+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804c570 con 0x7fb8000ff360 2024-03-12T03:25:53.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.927+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001be70 tx=0x7fb7f8004b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:53.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001bea0 con 0x7fb8000ff360 2024-03-12T03:25:53.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00455a0 con 0x7fb8000ff360 2024-03-12T03:25:53.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:53.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059070 con 0x7fb8000ff360 2024-03-12T03:25:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:54.234+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:54.234+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:54.234+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:54.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:54.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:54.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:54.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:54.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:54.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:55.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:55.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:55.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:55.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:55.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:55.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:55.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:55.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:55.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:56.462 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:56.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:56.462 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:56.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:56.462 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:56.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:56.463 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:56.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:56.463 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:56.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:56.463 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:56.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:56.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:56.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:56.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:57.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:57.289+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.177:6828/588231664, fault. 2024-03-12T03:25:57.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:57.289+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-12T03:25:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:57.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:57.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:57.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:57.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:57.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:57.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:57.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:57.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:57.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:58.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:58.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:58.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:58.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:58.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:58.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:58.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:58.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:58.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:58.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-12T03:25:58.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:25:58.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001be70 tx=0x7fb7f8004b80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001be70 tx=0x7fb7f8004b80 comp rx=0 tx=0).stop 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c3a0 con 0x7fb7e8040cc0 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.926+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c4e0 con 0x7fb8000ffcc0 2024-03-12T03:25:58.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:25:58.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:25:58.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:25:58.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804c620 con 0x7fb7e8040cc0 2024-03-12T03:25:58.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.928+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800cd40 tx=0x7fb7f004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:25:58.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f800de70 con 0x7fb7e8040cc0 2024-03-12T03:25:58.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f80490d0 con 0x7fb7e8040cc0 2024-03-12T03:25:58.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:25:58.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8003ac0 con 0x7fb7e8040cc0 2024-03-12T03:25:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:59.324+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:59.324+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:25:59.324+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:25:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:59.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:59.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:25:59.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:25:59.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:59.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:59.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:59.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:25:59.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:25:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:25:59.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:00.339+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:00.339+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:00.339+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:00.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:00.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:00.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:00.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:00.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:00.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:01.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:01.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:01.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:01.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:01.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:01.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:01.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:01.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:01.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:02.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:02.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:02.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:02.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:02.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:02.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:02.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:02.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:02.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:03.516 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:03.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:03.516 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:03.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:03.516 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:03.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:03.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:03.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:03.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:03.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:03.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:03.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:03.517 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:03.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:03.517 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:03.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:03.517 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:03.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800cd40 tx=0x7fb7f004c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800cd40 tx=0x7fb7f004c000 comp rx=0 tx=0).stop 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c860 con 0x7fb8000ff360 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c9a0 con 0x7fb8000ffcc0 2024-03-12T03:26:03.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.928+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.928+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.928+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804cae0 con 0x7fb8000ff360 2024-03-12T03:26:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.928+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8010040 tx=0x7fb7f8029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f800de70 con 0x7fb8000ff360 2024-03-12T03:26:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009270 con 0x7fb8000ff360 2024-03-12T03:26:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:03.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80459b0 con 0x7fb8000ff360 2024-03-12T03:26:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:04.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:04.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:04.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:04.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:04.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:04.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:04.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:04.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:04.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:04.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:04.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:04.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:05.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:05.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:05.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:05.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:05.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:05.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:05.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:05.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:05.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:06.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:06.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:06.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:06.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:06.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:06.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:06.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:06.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:06.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:07.505 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:07.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:07.505 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:07.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:07.505 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:07.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:07.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:07.293+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:07.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:07.293+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:07.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:07.293+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:07.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:07.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:07.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:07.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:08.480 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:08.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:08.480 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:08.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:08.480 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:08.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:08.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:08.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:08.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:08.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:08.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:08.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:08.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:08.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:08.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:26:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8010040 tx=0x7fb7f8029000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:08.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8010040 tx=0x7fb7f8029000 comp rx=0 tx=0).stop 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e8040cc0 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cdb0 con 0x7fb8000ffcc0 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.928+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.928+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:08.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.928+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804cef0 con 0x7fb7e8040cc0 2024-03-12T03:26:08.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.928+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80408c0 tx=0x7fb7f8047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:08.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801dc50 con 0x7fb7e8040cc0 2024-03-12T03:26:08.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8015420 con 0x7fb7e8040cc0 2024-03-12T03:26:08.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:08.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80081f0 con 0x7fb7e8040cc0 2024-03-12T03:26:09.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:09.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:09.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:09.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:09.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:09.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:09.475 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:09.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:09.475 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:09.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:09.475 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:09.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:09.657 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:09.655+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:26:09.657 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:09.655+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-12T03:26:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:09.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:09.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:09.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:10.437 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:10.330+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:10.437 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:10.330+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:10.437 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:10.330+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:10.438 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:10.141+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:10.438 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:10.141+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:10.438 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:10.141+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:10.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:10.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:10.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:11.443 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:11.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:11.444 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:11.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:11.444 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:11.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:11.444 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:11.139+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:11.444 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:11.139+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:11.444 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:11.139+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:11.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:11.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:11.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:12.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:12.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:12.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:12.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:12.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:12.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:12.412 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:12.093+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:12.412 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:12.093+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:12.412 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:12.093+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:12.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:12.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:12.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:13.431 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:13.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:13.431 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:13.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:13.431 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:13.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:13.432 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:13.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:13.432 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:13.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:13.432 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:13.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:13.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:13.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:13.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:13.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:26:13.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:13.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80408c0 tx=0x7fb7f8047040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80408c0 tx=0x7fb7f8047040 comp rx=0 tx=0).stop 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb8000ff360 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.927+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cdb0 con 0x7fb8000ffcc0 2024-03-12T03:26:13.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.928+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:13.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.928+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:13.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.928+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:13.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.928+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804d080 con 0x7fb8000ff360 2024-03-12T03:26:13.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005af90 tx=0x7fb7f801ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:13.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0012ed0 con 0x7fb8000ff360 2024-03-12T03:26:13.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004c3d0 con 0x7fb8000ff360 2024-03-12T03:26:13.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:13.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb8000ff360 2024-03-12T03:26:14.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:14.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:14.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:14.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:14.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:14.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:14.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:14.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:14.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:14.442+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:14.442+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:14.442+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:15.437 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:15.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:15.437 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:15.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:15.437 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:15.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:15.438 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:15.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:15.438 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:15.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:15.438 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:15.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:15.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:15.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:15.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:15.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:15.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:15.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:16.441 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:16.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:16.441 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:16.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:16.441 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:16.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:16.441 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:16.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:16.441 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:16.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:16.441 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:16.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:16.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:16.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:16.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:17.471 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:17.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:17.472 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:17.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:17.472 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:17.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:17.472 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:17.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:17.472 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:17.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:17.472 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:17.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:17.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:17.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:17.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:18.444 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:18.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:18.444 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:18.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:18.444 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:18.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:18.445 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:18.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:18.445 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:18.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:18.445 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:18.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:18.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:18.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:18.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:18.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.871+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:26:18.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.871+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005af90 tx=0x7fb7f801ff90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005af90 tx=0x7fb7f801ff90 comp rx=0 tx=0).stop 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:18.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e8040cc0 2024-03-12T03:26:18.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cdb0 con 0x7fb8000ffcc0 2024-03-12T03:26:18.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:18.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.929+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:18.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:18.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.929+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804d080 con 0x7fb7e8040cc0 2024-03-12T03:26:18.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb7f002cf80 tx=0x7fb7f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:18.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001bea0 con 0x7fb7e8040cc0 2024-03-12T03:26:18.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001be70 con 0x7fb7e8040cc0 2024-03-12T03:26:18.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:18.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059050 con 0x7fb7e8040cc0 2024-03-12T03:26:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:19.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:19.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:19.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:19.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:19.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:19.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:19.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:19.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:19.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:19.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:19.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:19.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:20.436 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:20.142+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:20.436 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:20.142+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:20.436 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:20.142+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:20.436 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:20.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:20.436 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:20.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:20.436 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:20.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:20.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:20.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:20.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:21.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:21.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:21.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:21.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:21.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:21.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:21.474 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:21.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:21.474 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:21.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:21.474 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:21.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:21.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:21.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:21.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:22.496 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:22.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:22.497 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:22.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:22.497 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:22.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:22.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:22.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:22.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:22.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:22.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:22.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:22.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:22.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:22.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:23.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:23.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:23.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:23.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:23.518 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:23.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:23.518 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:23.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:23.518 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:23.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:23.518 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:23.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:23.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:23.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:23.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:23.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:23.519 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:23.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:23.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:26:23.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:23.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb7f002cf80 tx=0x7fb7f000b890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:23.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fb7f002cf80 tx=0x7fb7f000b890 comp rx=0 tx=0).stop 2024-03-12T03:26:23.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:23.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:23.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:23.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:23.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb8000ff360 2024-03-12T03:26:23.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.928+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cdb0 con 0x7fb8000ffcc0 2024-03-12T03:26:23.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.929+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:23.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.929+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:23.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.929+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:23.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.929+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804d080 con 0x7fb8000ff360 2024-03-12T03:26:23.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.930+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8008000 tx=0x7fb7f0038070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:23.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f800c760 con 0x7fb8000ff360 2024-03-12T03:26:23.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800cec0 con 0x7fb8000ff360 2024-03-12T03:26:23.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:23.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f801fbe0 con 0x7fb8000ff360 2024-03-12T03:26:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:24.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:24.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:24.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:24.262+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:24.262+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:24.262+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:24.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:24.249+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:24.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:24.249+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:24.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:24.249+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:25.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:25.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:25.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:25.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:25.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:25.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:25.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:25.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:25.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:25.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:25.467+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:25.467+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:25.467+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:26.472 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:26.211+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:26.473 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:26.211+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:26.473 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:26.211+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:26.473 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:26.189+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:26.473 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:26.189+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:26.473 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:26.189+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:26.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:26.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:26.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:27.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:27.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:27.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:27.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:27.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:27.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:27.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:27.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:27.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:27.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:28.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:28.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:28.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:28.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:28.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:28.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:28.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:28.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:28.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.928+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.928+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.928+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8008000 tx=0x7fb7f0038070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8008000 tx=0x7fb7f0038070 comp rx=0 tx=0).stop 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c030 con 0x7fb7e8040cc0 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb8000ffcc0 2024-03-12T03:26:28.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:28.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:28.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:28.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804d210 con 0x7fb7e8040cc0 2024-03-12T03:26:28.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f470 tx=0x7fb7f8045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:28.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0012ed0 con 0x7fb7e8040cc0 2024-03-12T03:26:28.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0044e40 con 0x7fb7e8040cc0 2024-03-12T03:26:28.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:28.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8040cc0 2024-03-12T03:26:29.280 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:29.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:29.280 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:29.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:29.280 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:29.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:29.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:29.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:29.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:29.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:29.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:29.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:30.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:30.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:30.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:30.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:30.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:30.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:30.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:30.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:30.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:31.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:31.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:31.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:31.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:31.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:31.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:31.397 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:31.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:31.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:31.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:31.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:31.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:31.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:31.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:31.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:32.396 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:32.111+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:32.396 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:32.111+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:32.396 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:32.111+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:32.396 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:32.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:32.396 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:32.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:32.396 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:32.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:32.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:32.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:32.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:33.518 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:33.147+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:33.518 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:33.147+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:33.518 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:33.147+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:33.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:33.234+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:33.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:33.234+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:33.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:33.234+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:33.520 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:33.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:33.520 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:33.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:33.520 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:33.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:33.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:26:33.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:33.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f470 tx=0x7fb7f8045000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:33.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f470 tx=0x7fb7f8045000 comp rx=0 tx=0).stop 2024-03-12T03:26:33.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 msgr2=0x7fb7e8049c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:33.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8040cc0 0x7fb7e8049c50 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:33.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:33.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:33.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804d210 con 0x7fb8000ff360 2024-03-12T03:26:33.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804d350 con 0x7fb8000ffcc0 2024-03-12T03:26:33.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:33.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:33.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:33.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804d490 con 0x7fb8000ff360 2024-03-12T03:26:33.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0063010 tx=0x7fb7f00090e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:33.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045640 con 0x7fb8000ff360 2024-03-12T03:26:33.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0044e40 con 0x7fb8000ff360 2024-03-12T03:26:33.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:33.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb8000ff360 2024-03-12T03:26:34.233 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:34.231+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:26:34.233 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:34.231+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:26:34.234 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:34.154+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:34.234 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:34.154+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:34.234 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:34.154+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:34.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:34.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:34.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:34.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:34.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:34.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:35.421 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:35.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:35.421 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:35.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:35.421 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:35.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:35.421 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:35.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:35.421 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:35.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:35.422 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:35.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:35.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:35.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:35.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:36.439 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:36.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:36.439 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:36.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:36.439 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:36.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:36.439 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:36.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:36.439 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:36.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:36.439 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:36.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:36.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:36.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:36.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:37.411 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:37.205+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:37.411 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:37.205+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:37.411 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:37.205+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:37.411 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:37.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:37.412 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:37.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:37.412 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:37.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:37.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:37.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:37.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:38.436 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:38.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:38.437 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:38.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:38.437 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:38.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:38.437 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:38.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:38.437 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:38.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:38.437 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:38.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:38.807 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:38.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:38.807 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:38.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:38.807 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:38.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:38.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:26:38.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:38.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0063010 tx=0x7fb7f00090e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:38.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0063010 tx=0x7fb7f00090e0 comp rx=0 tx=0).stop 2024-03-12T03:26:38.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 msgr2=0x7fb7e8045a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:38.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb8000ff360 0x7fb7e8045a20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:38.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:38.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:38.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804a030 con 0x7fb7e8049c50 2024-03-12T03:26:38.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.929+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804d750 con 0x7fb8000ffcc0 2024-03-12T03:26:38.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:38.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:38.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:38.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804d890 con 0x7fb7e8049c50 2024-03-12T03:26:38.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb7f002c930 tx=0x7fb7f002c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045640 con 0x7fb7e8049c50 2024-03-12T03:26:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0044e40 con 0x7fb7e8049c50 2024-03-12T03:26:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:38.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8049c50 2024-03-12T03:26:39.406 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:39.148+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:39.406 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:39.148+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:39.406 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:39.148+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:39.407 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:39.104+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:39.407 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:39.104+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:39.407 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:39.104+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:39.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:39.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:39.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:40.378 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:40.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:40.378 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:40.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:40.378 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:40.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:40.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:40.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:40.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:40.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:40.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:40.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:40.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:40.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:40.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:41.384 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:41.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:41.384 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:41.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:41.384 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:41.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:41.385 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:41.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:41.385 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:41.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:41.385 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:41.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:41.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:41.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:41.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:42.360 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:42.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:42.360 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:42.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:42.360 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:42.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:42.360 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:42.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:42.360 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:42.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:42.360 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:42.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:42.357+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:42.357+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:42.357+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:43.405 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:43.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:43.405 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:43.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:43.405 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:43.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:43.406 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:43.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:43.406 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:43.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:43.406 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:43.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:43.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:43.825 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:43.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:43.825 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:43.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:43.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb7f002c930 tx=0x7fb7f002c990 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb7f002c930 tx=0x7fb7f002c990 comp rx=0 tx=0).stop 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 0x7fb7e804d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804dc90 con 0x7fb7e8043a10 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804ddd0 con 0x7fb8000ffcc0 2024-03-12T03:26:43.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 0x7fb7e804d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:43.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:43.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:43.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804df10 con 0x7fb7e8043a10 2024-03-12T03:26:43.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 0x7fb7e804d750 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b8d0 tx=0x7fb7f0044fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:43.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0012ed0 con 0x7fb7e8043a10 2024-03-12T03:26:43.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0035430 con 0x7fb7e8043a10 2024-03-12T03:26:43.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:43.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059070 con 0x7fb7e8043a10 2024-03-12T03:26:44.193 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:44.093+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:44.193 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:44.093+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:44.193 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:44.093+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:44.367+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:44.367+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:44.367+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:44.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:44.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:44.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:45.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:45.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:45.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:45.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:45.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:45.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:45.357 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:45.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:45.358 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:45.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:45.358 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:45.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:45.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:45.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:45.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:46.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:46.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:46.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:46.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:46.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:46.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:46.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:46.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:46.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:46.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:46.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:46.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:46.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:46.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:46.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:47.362 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:47.247+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:47.362 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:47.247+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:47.362 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:47.247+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:47.363 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:47.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:47.363 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:47.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:47.363 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:47.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:47.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:47.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:47.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:48.403 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:48.090+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:48.403 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:48.090+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:48.403 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:48.090+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:48.404 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:48.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:48.404 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:48.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:48.404 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:48.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:48.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:48.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:48.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 msgr2=0x7fb7e804d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 msgr2=0x7fb7e804d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 0x7fb7e804d750 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b8d0 tx=0x7fb7f0044fe0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 0x7fb7e804d750 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b8d0 tx=0x7fb7f0044fe0 comp rx=0 tx=0).stop 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 msgr2=0x7fb7e804d750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043a10 0x7fb7e804d750 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804dc90 con 0x7fb7e8049c50 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804ddd0 con 0x7fb8000ffcc0 2024-03-12T03:26:48.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:48.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:48.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:48.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804df10 con 0x7fb7e8049c50 2024-03-12T03:26:48.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000de60 tx=0x7fb7f000bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:48.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0012ed0 con 0x7fb7e8049c50 2024-03-12T03:26:48.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0035430 con 0x7fb7e8049c50 2024-03-12T03:26:48.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:48.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8049c50 2024-03-12T03:26:49.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:49.085+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:49.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:49.085+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:49.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:49.085+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:49.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:49.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:49.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:49.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:49.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:49.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:49.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:49.591+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:26:49.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:49.591+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:26:50.338 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:50.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:50.338 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:50.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:50.338 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:50.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:50.338 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:50.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:50.338 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:50.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:50.339 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:50.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:50.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:50.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:50.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:51.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:51.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:51.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:51.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:51.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:51.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:51.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:51.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:51.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:52.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:52.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:52.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:52.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:52.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:52.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:52.270+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:52.270+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:52.270+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:53.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:53.301+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:53.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:53.301+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:53.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:53.301+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:53.519 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:53.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:53.519 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:53.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:53.519 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:53.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:53.520 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:53.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:53.520 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:53.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:53.520 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:53.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:53.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:26:53.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:53.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000de60 tx=0x7fb7f000bc90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:53.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000de60 tx=0x7fb7f000bc90 comp rx=0 tx=0).stop 2024-03-12T03:26:53.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:53.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:53.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 0x7fb7e80441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:53.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:53.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044710 con 0x7fb7e8043df0 2024-03-12T03:26:53.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044850 con 0x7fb8000ffcc0 2024-03-12T03:26:53.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 0x7fb7e80441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:53.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:53.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:53.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8044990 con 0x7fb7e8043df0 2024-03-12T03:26:53.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 0x7fb7e80441d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0051ff0 tx=0x7fb7f004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:53.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0012ed0 con 0x7fb7e8043df0 2024-03-12T03:26:53.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00455a0 con 0x7fb7e8043df0 2024-03-12T03:26:53.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:53.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8043df0 2024-03-12T03:26:54.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:54.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:54.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:54.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:54.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:54.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:54.228+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:54.228+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:54.228+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:54.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:54.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:54.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:55.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:55.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:55.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:55.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:55.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:55.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:55.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:55.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:55.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:56.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:56.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:56.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:56.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:56.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:56.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:56.350+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:56.350+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:56.350+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:57.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:57.000+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:57.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:57.000+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:57.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:57.000+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:57.300+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:57.300+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:57.300+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:57.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:57.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:57.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:57.992+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:57.992+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:57.992+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:58.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:58.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:58.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:58.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:58.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:58.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:58.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 msgr2=0x7fb7e80441d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:26:58.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 msgr2=0x7fb7e80441d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:26:58.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 0x7fb7e80441d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0051ff0 tx=0x7fb7f004c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:26:58.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 0x7fb7e80441d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0051ff0 tx=0x7fb7f004c000 comp rx=0 tx=0).stop 2024-03-12T03:26:58.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 msgr2=0x7fb7e80441d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:26:58.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8043df0 0x7fb7e80441d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:58.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:58.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:26:58.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044710 con 0x7fb7e8049c50 2024-03-12T03:26:58.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.930+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044850 con 0x7fb8000ffcc0 2024-03-12T03:26:58.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:26:58.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:26:58.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:26:58.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8044990 con 0x7fb7e8049c50 2024-03-12T03:26:58.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00081d0 tx=0x7fb7f0009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:26:58.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d7b0 con 0x7fb7e8049c50 2024-03-12T03:26:58.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001bde0 con 0x7fb7e8049c50 2024-03-12T03:26:58.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:26:58.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000f6d0 con 0x7fb7e8049c50 2024-03-12T03:26:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:59.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:59.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:26:59.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:26:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:59.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:59.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:26:59.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:26:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:59.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:59.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:26:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:26:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:26:59.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:00.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:00.064+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:00.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:00.064+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:00.394 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:00.064+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:00.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:00.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:00.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:00.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:00.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:00.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:01.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:01.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:01.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:01.390+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:01.390+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:01.390+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:01.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:01.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:01.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:02.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:02.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:02.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:02.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:02.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:02.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:02.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:02.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:02.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:02.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:02.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:02.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:03.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:03.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:03.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:03.334+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:03.334+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:03.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:03.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:27:03.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:03.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00081d0 tx=0x7fb7f0009900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:03.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00081d0 tx=0x7fb7f0009900 comp rx=0 tx=0).stop 2024-03-12T03:27:03.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:03.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 0x7fb7e80445b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044d60 con 0x7fb7e80441d0 2024-03-12T03:27:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044ea0 con 0x7fb8000ffcc0 2024-03-12T03:27:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 0x7fb7e80445b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:03.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:03.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:03.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8044fe0 con 0x7fb7e80441d0 2024-03-12T03:27:03.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 0x7fb7e80445b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0020f90 tx=0x7fb7f004cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d7b0 con 0x7fb7e80441d0 2024-03-12T03:27:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0035430 con 0x7fb7e80441d0 2024-03-12T03:27:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:03.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e80441d0 2024-03-12T03:27:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:03.931+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:03.987+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:03.987+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:03.987+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:04.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:04.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:04.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:04.355+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:04.355+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:04.355+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:04.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:04.952+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:27:04.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:04.952+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:27:05.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:04.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:05.292 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:04.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:05.292 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:04.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:05.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:05.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:05.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:05.369+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:05.369+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:05.369+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:06.253 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:05.938+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:06.253 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:05.938+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:06.253 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:05.938+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:06.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:06.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:06.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:06.250+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:06.250+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:06.250+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:07.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:06.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:07.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:06.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:07.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:06.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:07.382+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:07.382+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:07.382+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:07.293+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:07.293+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:07.293+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:08.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:07.903+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:08.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:07.903+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:08.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:07.903+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:08.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:08.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:08.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:08.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:08.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:08.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:08.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 msgr2=0x7fb7e80445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:27:08.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 msgr2=0x7fb7e80445b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:08.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 0x7fb7e80445b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0020f90 tx=0x7fb7f004cd10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:08.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 0x7fb7e80445b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0020f90 tx=0x7fb7f004cd10 comp rx=0 tx=0).stop 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 msgr2=0x7fb7e80445b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80441d0 0x7fb7e80445b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044e30 con 0x7fb7e8049c50 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.931+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044f70 con 0x7fb8000ffcc0 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:08.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80450b0 con 0x7fb7e8049c50 2024-03-12T03:27:08.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004ff90 tx=0x7fb7f005b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:08.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d7b0 con 0x7fb7e8049c50 2024-03-12T03:27:08.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0035430 con 0x7fb7e8049c50 2024-03-12T03:27:08.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:08.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059050 con 0x7fb7e8049c50 2024-03-12T03:27:09.277 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:08.931+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:09.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:08.934+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:09.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:08.934+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:09.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:08.934+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:09.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:09.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:09.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:09.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:09.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:09.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:10.288 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:09.969+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:10.288 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:09.969+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:10.288 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:09.969+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:10.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:10.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:10.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:10.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:10.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:10.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:11.270 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:10.977+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:11.270 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:10.977+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:11.270 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:10.977+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:11.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:11.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:11.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:11.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:11.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:11.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:12.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:11.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:12.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:11.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:12.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:11.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:12.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:12.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:12.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:12.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:12.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:12.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:13.242 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:12.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:13.243 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:12.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:13.243 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:12.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:13.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:13.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:13.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:13.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:13.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:13.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:13.521 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:13.240+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:13.521 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:13.240+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:13.522 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:13.240+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:13.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:27:13.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.931+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:13.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004ff90 tx=0x7fb7f005b7f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:13.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.931+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004ff90 tx=0x7fb7f005b7f0 comp rx=0 tx=0).stop 2024-03-12T03:27:13.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:13.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8044a60 con 0x7fb7e80445b0 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.933+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.933+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:13.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.933+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e80445b0 2024-03-12T03:27:13.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.933+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009160 tx=0x7fb7f000d9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:13.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f00459f0 con 0x7fb7e80445b0 2024-03-12T03:27:13.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0035430 con 0x7fb7e80445b0 2024-03-12T03:27:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:13.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e80445b0 2024-03-12T03:27:14.206 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:13.932+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 31 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:14.206 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:13.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:14.206 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:13.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:14.206 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:13.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:14.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:14.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:14.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:14.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:14.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:14.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:15.212 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:14.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:15.212 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:14.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:15.213 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:14.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:15.209+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:15.209+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:15.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:15.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:15.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:15.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:16.195 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:15.924+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:16.195 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:15.924+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:16.195 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:15.924+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:16.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:16.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:16.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:16.226+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:16.226+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:16.226+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:17.186 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:16.928+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:17.186 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:16.928+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:17.186 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:16.928+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:17.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:17.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:17.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:17.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:17.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:17.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:18.185 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:17.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:18.185 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:17.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:18.185 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:17.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:18.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:18.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:18.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:18.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:18.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:18.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009160 tx=0x7fb7f000d9a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009160 tx=0x7fb7f000d9a0 comp rx=0 tx=0).stop 2024-03-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:18.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb7e8049c50 2024-03-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80454b0 con 0x7fb8000ffcc0 2024-03-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.933+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:18.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.933+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.933+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.933+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80455f0 con 0x7fb7e8049c50 2024-03-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.934+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000ec90 tx=0x7fb7f000bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000ecc0 con 0x7fb7e8049c50 2024-03-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000f500 con 0x7fb7e8049c50 2024-03-12T03:27:18.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:18.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059050 con 0x7fb7e8049c50 2024-03-12T03:27:19.264 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:18.932+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 50 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:19.264 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:19.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:19.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:19.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:19.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:19.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:19.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:18.939+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:18.939+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:19.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:18.939+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:20.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:20.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:20.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:19.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:19.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:19.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:20.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:20.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:20.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:20.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:20.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:20.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:20.311+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:27:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:20.311+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:27:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:21.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:21.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:21.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:21.142+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:21.142+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:21.142+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:21.232+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:21.232+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:21.232+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:22.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:22.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:22.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:22.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:22.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:22.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:22.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:22.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:22.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:22.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:22.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:22.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:23.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:23.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:23.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:23.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:23.074+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:23.074+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:23.074+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:23.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:23.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:23.522 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:23.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:23.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:27:23.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:23.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000ec90 tx=0x7fb7f000bc90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:23.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000ec90 tx=0x7fb7f000bc90 comp rx=0 tx=0).stop 2024-03-12T03:27:23.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:23.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b420 con 0x7fb7e80445b0 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.932+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e80445b0 2024-03-12T03:27:23.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804c960 tx=0x7fb7f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:23.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80490d0 con 0x7fb7e80445b0 2024-03-12T03:27:23.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800b8d0 con 0x7fb7e80445b0 2024-03-12T03:27:23.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:23.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8003ac0 con 0x7fb7e80445b0 2024-03-12T03:27:24.203 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:23.932+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 66 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:24.204 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:24.106+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:24.204 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:24.106+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:24.204 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:24.106+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:24.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:24.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:24.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:24.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:24.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:24.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:25.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:25.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:25.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:25.084+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:25.084+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:25.084+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:25.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:25.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:25.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:26.457 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:26.081+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:26.458 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:26.081+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:26.458 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:26.081+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:26.458 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:26.172+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:26.458 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:26.172+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:26.458 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:26.172+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:26.459 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:26.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:26.459 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:26.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:26.459 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:26.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:27.052+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:27.052+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:27.052+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:27.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:27.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:27.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:27.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:27.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:27.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:28.088+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:28.088+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:28.088+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:28.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:28.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:28.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:28.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:28.262+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:28.262+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:28.262+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.932+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.932+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.932+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804c960 tx=0x7fb7f000b890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.932+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804c960 tx=0x7fb7f000b890 comp rx=0 tx=0).stop 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:28.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b420 con 0x7fb7e8049c50 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e8049c50 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.934+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800fb90 tx=0x7fb7f800c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80084f0 con 0x7fb7e8049c50 2024-03-12T03:27:28.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f80081f0 con 0x7fb7e8049c50 2024-03-12T03:27:28.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:28.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c760 con 0x7fb7e8049c50 2024-03-12T03:27:29.191 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:28.932+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 84 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:29.191 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:29.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:29.191 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:29.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:29.191 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:29.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:29.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:29.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:29.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:29.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:29.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:29.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:30.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:30.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:30.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:30.141+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:30.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:30.141+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:30.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:30.141+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:30.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:30.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:30.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:31.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:31.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:31.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:31.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:31.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:31.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:31.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:31.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:31.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:32.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:32.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:32.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:32.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:32.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:32.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:32.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:32.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:32.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:32.196+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:32.196+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:32.196+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:33.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:33.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:33.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:33.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:33.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:33.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:33.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:33.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:33.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:33.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:33.244+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:33.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:33.244+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:33.314 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:33.244+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:33.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:27:33.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:33.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800fb90 tx=0x7fb7f800c380 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:33.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800fb90 tx=0x7fb7f800c380 comp rx=0 tx=0).stop 2024-03-12T03:27:33.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:33.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:33.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:33.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:33.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b420 con 0x7fb7e80445b0 2024-03-12T03:27:33.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:27:33.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.934+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:33.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.934+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:33.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.934+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:33.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.934+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e80445b0 2024-03-12T03:27:33.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8029410 tx=0x7fb7f8032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:33.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804c730 con 0x7fb7e80445b0 2024-03-12T03:27:33.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f804c730 con 0x7fb7e80445b0 2024-03-12T03:27:33.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:33.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c760 con 0x7fb7e80445b0 2024-03-12T03:27:34.276 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:33.933+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 104 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:34.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:34.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:34.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:34.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:34.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:34.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:34.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:34.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:34.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:34.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:34.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:34.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:34.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:34.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:34.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:34.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:34.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:34.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:35.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:35.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:35.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:35.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:35.671+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:27:35.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:35.671+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:27:35.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:35.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:35.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:35.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:35.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:35.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:36.015+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:36.015+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:36.015+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:36.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:36.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:36.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:36.311+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:36.311+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:36.311+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:37.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:37.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:37.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:37.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:37.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:37.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:37.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:37.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:37.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:37.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:37.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:37.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:37.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:37.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:37.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:38.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:38.017+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:38.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:38.017+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:38.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:38.017+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:38.292 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:38.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:38.292 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:38.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:38.292 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:38.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:38.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:38.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:38.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8029410 tx=0x7fb7f8032000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8029410 tx=0x7fb7f8032000 comp rx=0 tx=0).stop 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b420 con 0x7fb7e8049c50 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:27:38.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.934+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:38.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:38.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:38.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e8049c50 2024-03-12T03:27:38.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802b040 tx=0x7fb7f8021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:38.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e8049c50 2024-03-12T03:27:38.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e8049c50 2024-03-12T03:27:38.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:38.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8008d40 con 0x7fb7e8049c50 2024-03-12T03:27:39.056 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:38.933+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 121 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:39.056 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:38.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:39.056 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:38.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:39.056 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:38.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:39.056 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:39.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:39.057 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:39.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:39.057 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:39.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:39.296+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:39.296+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:39.296+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:40.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:40.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:40.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:39.965+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:39.965+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:39.965+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:40.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:40.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:40.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:41.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:41.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:41.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:40.978+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:40.978+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:40.978+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:41.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:41.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:41.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:42.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:42.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:42.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:42.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:42.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:42.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:42.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:42.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:42.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:42.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:43.439 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:43.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:43.439 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:43.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:43.440 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:43.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:43.440 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:43.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:43.440 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:43.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:43.440 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:43.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:43.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:43.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:43.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802b040 tx=0x7fb7f8021040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802b040 tx=0x7fb7f8021040 comp rx=0 tx=0).stop 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:43.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b420 con 0x7fb7e80445b0 2024-03-12T03:27:43.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:27:43.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:43.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:43.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:43.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.933+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e80445b0 2024-03-12T03:27:43.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.934+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004dfb0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:43.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e80445b0 2024-03-12T03:27:43.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000f980 con 0x7fb7e80445b0 2024-03-12T03:27:43.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:43.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e80445b0 2024-03-12T03:27:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:43.933+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 137 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:44.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:44.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:44.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:44.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:44.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:44.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:44.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:44.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:44.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:45.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:45.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:45.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:45.070+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:45.070+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:45.070+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:45.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:45.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:45.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:46.033+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:46.033+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:46.033+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:46.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:46.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:46.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:46.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:46.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:46.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:46.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:46.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:47.105+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:47.105+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:47.105+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:47.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:47.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:47.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:47.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:47.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:47.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:48.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:48.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:48.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:48.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:48.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:48.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:48.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:48.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:48.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004dfb0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004dfb0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804b420 con 0x7fb7e8049c50 2024-03-12T03:27:48.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:27:48.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.935+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:48.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.935+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:48.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.935+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:48.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.935+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e8049c50 2024-03-12T03:27:48.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.935+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009160 tx=0x7fb7f004cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:48.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001ba90 con 0x7fb7e8049c50 2024-03-12T03:27:48.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004c760 con 0x7fb7e8049c50 2024-03-12T03:27:48.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:48.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000f6d0 con 0x7fb7e8049c50 2024-03-12T03:27:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:48.934+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 156 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:49.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:49.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:49.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:49.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:49.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:49.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:49.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:49.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:49.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:50.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:50.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:50.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:50.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:50.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:50.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:50.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:50.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:50.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:50.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:50.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:50.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:51.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:51.031+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:27:51.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:51.031+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:27:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:51.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:51.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:51.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:51.048+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:51.048+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:51.048+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:51.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:51.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:51.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:52.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:52.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:52.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:51.992+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:51.992+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:51.992+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:52.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:52.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:52.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:53.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:53.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:53.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:52.946+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:52.946+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:52.946+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:53.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:53.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:53.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:53.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:27:53.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:53.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009160 tx=0x7fb7f004cd10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:53.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0009160 tx=0x7fb7f004cd10 comp rx=0 tx=0).stop 2024-03-12T03:27:53.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:53.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb7e80445b0 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.934+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80454b0 con 0x7fb8000ffcc0 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80455f0 con 0x7fb7e80445b0 2024-03-12T03:27:53.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.936+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80022f0 tx=0x7fb7f0047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:53.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f800cd40 con 0x7fb7e80445b0 2024-03-12T03:27:53.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800cd40 con 0x7fb7e80445b0 2024-03-12T03:27:53.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:53.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c710 con 0x7fb7e80445b0 2024-03-12T03:27:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:53.934+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 174 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:54.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:54.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:54.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:54.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:53.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:54.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:53.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:54.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:53.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:54.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:54.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:54.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:54.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:54.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:54.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:55.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:55.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:55.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:55.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:55.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:55.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:55.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:55.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:55.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:56.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:56.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:56.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:56.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:56.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:56.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:57.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:57.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:57.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:57.025+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:57.025+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:57.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:57.025+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:57.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:57.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:57.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:57.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:57.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:57.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:58.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:58.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:58.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:58.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:58.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:58.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:58.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:58.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.934+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:27:58.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:27:58.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80022f0 tx=0x7fb7f0047040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:27:58.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80022f0 tx=0x7fb7f0047040 comp rx=0 tx=0).stop 2024-03-12T03:27:58.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb7e8049c50 2024-03-12T03:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80454b0 con 0x7fb8000ffcc0 2024-03-12T03:27:58.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.936+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.936+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.936+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80455f0 con 0x7fb7e8049c50 2024-03-12T03:27:58.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.936+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80187a0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:27:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8049740 con 0x7fb7e8049c50 2024-03-12T03:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8049740 con 0x7fb7e8049c50 2024-03-12T03:27:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:27:58.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8030d30 con 0x7fb7e8049c50 2024-03-12T03:27:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:27:58.935+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 190 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:27:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:58.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:58.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:58.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:27:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:58.950+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:27:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:59.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:59.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:59.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:27:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:27:59.017+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:27:59.710 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:59.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:59.710 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:59.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:59.710 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:59.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:27:59.710 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:27:59.418+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:00.020 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:59.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:00.020 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:59.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:00.020 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:59.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:00.020 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:27:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:27:59.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:00.017+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:00.017+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:00.017+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:00.018+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:00.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:00.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:00.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:00.417+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:01.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:01.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:01.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:01.037+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:00.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:00.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:00.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:01.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:00.926+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:01.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:01.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:01.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:01.402+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:02.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:02.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:02.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:02.037+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:01.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:01.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:01.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:01.908+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:02.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:02.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:02.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:02.360+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:03.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:03.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:03.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:03.009+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:02.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:02.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:02.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:02.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:03.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:03.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:03.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:03.352+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80187a0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80187a0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb7e80445b0 2024-03-12T03:28:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80454b0 con 0x7fb8000ffcc0 2024-03-12T03:28:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.935+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.936+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.936+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80455f0 con 0x7fb7e80445b0 2024-03-12T03:28:03.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046c80 tx=0x7fb7f8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:03.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e80445b0 2024-03-12T03:28:03.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f002de40 con 0x7fb7e80445b0 2024-03-12T03:28:03.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:03.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e80445b0 2024-03-12T03:28:04.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:04.258+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e0003550 con 0x7fb7e80445b0 2024-03-12T03:28:04.261 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:03.935+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 211 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:04.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:04.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:04.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:04.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:04.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:04.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:04.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:04.055+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:04.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:03.944+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:04.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:03.944+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:04.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:03.944+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:04.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:03.945+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:04.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:04.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:04.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:04.353+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:05.042 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:04.943+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:05.042 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:04.943+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:05.043 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:04.943+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:05.043 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:04.944+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:05.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:05.258+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e0005870 con 0x7fb7e80445b0 2024-03-12T03:28:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:05.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:05.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:05.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:05.040+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:05.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:05.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:05.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:05.393+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:06.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:06.258+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e0005870 con 0x7fb7e80445b0 2024-03-12T03:28:06.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:06.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:06.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:06.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:06.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:06.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:06.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:06.051+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:06.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:05.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:06.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:05.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:06.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:05.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:06.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:05.912+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:06.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:06.391+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:28:06.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:06.391+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:28:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:06.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:06.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:06.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:06.416+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:07.258+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e0005870 con 0x7fb7e80445b0 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:07.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:07.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:07.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:07.019+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:06.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:06.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:06.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:07.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:06.933+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:07.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:07.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:07.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:07.383+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:08.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.258+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e0005870 con 0x7fb7e80445b0 2024-03-12T03:28:08.261 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:08.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:08.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:08.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.011+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:08.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:07.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:08.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:07.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:08.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:07.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:08.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:07.959+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:08.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:08.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:08.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:08.411+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:08.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.935+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.935+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.935+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046c80 tx=0x7fb7f8000f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.935+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046c80 tx=0x7fb7f8000f40 comp rx=0 tx=0).stop 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.935+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.935+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb7e8049c50 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80454b0 con 0x7fb8000ffcc0 2024-03-12T03:28:08.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.936+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:08.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.937+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:08.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.937+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:08.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.937+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80455f0 con 0x7fb7e8049c50 2024-03-12T03:28:08.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.937+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801afb0 tx=0x7fb7f00099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:08.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f800d970 con 0x7fb7e8049c50 2024-03-12T03:28:08.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800d7a0 con 0x7fb7e8049c50 2024-03-12T03:28:08.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:08.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800db40 con 0x7fb7e8049c50 2024-03-12T03:28:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:08.935+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 228 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:08.997+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:08.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:08.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:08.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:08.953+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:09.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:09.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:09.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:09.366+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:09.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:09.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:09.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:09.957+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:09.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:09.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:09.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:09.989+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:10.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:10.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:10.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:10.405+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:11.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:11.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:11.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:11.028+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:10.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:10.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:10.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:10.964+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:11.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:11.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:11.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:11.433+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:12.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:12.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:12.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:12.026+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:11.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:11.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:11.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:12.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:11.970+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:12.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:12.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:12.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:12.416+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:12.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:12.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:12.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:12.947+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:13.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:13.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:13.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:13.061+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:13.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:13.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:13.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:13.462+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.935+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801afb0 tx=0x7fb7f00099b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801afb0 tx=0x7fb7f00099b0 comp rx=0 tx=0).stop 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb7e80445b0 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80454b0 con 0x7fb8000ffcc0 2024-03-12T03:28:13.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.936+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:13.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:13.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:13.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80455f0 con 0x7fb7e80445b0 2024-03-12T03:28:13.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0048f10 tx=0x7fb7f800c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:13.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e80445b0 2024-03-12T03:28:13.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0011c10 con 0x7fb7e80445b0 2024-03-12T03:28:13.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:13.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059070 con 0x7fb7e80445b0 2024-03-12T03:28:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:13.936+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 246 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:13.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:13.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:13.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:13.899+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:14.097+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:14.097+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:14.097+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:14.099+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:14.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:14.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:14.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:14.440+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:15.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:14.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:15.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:14.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:15.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:14.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:15.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:14.852+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:15.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:15.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:15.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:15.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:15.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:15.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:15.394 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:15.127+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:15.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:15.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:15.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:15.391+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:15.808+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:15.808+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:15.808+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:15.809+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:16.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:16.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:16.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:16.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:16.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:16.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:16.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:16.108+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:16.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:16.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:16.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:16.423+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:17.156 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:16.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:17.156 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:16.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:17.156 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:16.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:17.156 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:16.811+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:17.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:17.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:17.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:17.154+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:17.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:17.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:17.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:17.384+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:18.130 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:17.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:18.130 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:17.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:18.131 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:17.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:18.131 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:17.835+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:18.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:18.127+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:18.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:18.127+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:18.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:18.128+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:18.393 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:18.128+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:18.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:18.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:18.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:18.391+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0048f10 tx=0x7fb7f800c710 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0048f10 tx=0x7fb7f800c710 comp rx=0 tx=0).stop 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:18.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:18.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c190 con 0x7fb7e8049c50 2024-03-12T03:28:18.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:28:18.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.936+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:18.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:18.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:18.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e8049c50 2024-03-12T03:28:18.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0053610 tx=0x7fb7f0065020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:18.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e8049c50 2024-03-12T03:28:18.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0011c10 con 0x7fb7e8049c50 2024-03-12T03:28:18.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:18.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8049c50 2024-03-12T03:28:19.112 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:18.936+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 262 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:19.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:18.847+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:19.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:18.847+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:19.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:18.847+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:19.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:18.848+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:19.404 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:19.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:19.404 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:19.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:19.404 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:19.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:19.404 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:19.111+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:19.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:19.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:19.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:19.402+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:20.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:19.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:20.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:19.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:20.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:19.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:20.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:19.815+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:20.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:20.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:20.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:20.144+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:20.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:20.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:20.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:20.372+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:20.796+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:20.796+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:20.796+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:20.797+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:21.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:21.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:21.346+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:21.346+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:21.168+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:21.168+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:21.168+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:21.169+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:21.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:21.751+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:28:21.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:21.751+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:28:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:21.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:21.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:21.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:21.829+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:22.189+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:22.189+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:22.189+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:22.190+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:22.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:22.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:22.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:22.388+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:23.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:22.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:23.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:22.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:23.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:22.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:23.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:22.852+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:23.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:23.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:23.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:23.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:23.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:23.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:23.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:23.166+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:23.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:23.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:23.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:23.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:23.527 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:23.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:23.527 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:23.363+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:23.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:28:23.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:23.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0053610 tx=0x7fb7f0065020 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0053610 tx=0x7fb7f0065020 comp rx=0 tx=0).stop 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c190 con 0x7fb7e80445b0 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.936+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:28:23.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.937+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:23.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.937+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:23.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.937+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:23.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.937+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e80445b0 2024-03-12T03:28:23.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.938+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80022f0 tx=0x7fb7f004cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:23.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fe10 con 0x7fb7e80445b0 2024-03-12T03:28:23.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8049740 con 0x7fb7e80445b0 2024-03-12T03:28:23.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:23.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80493d0 con 0x7fb7e80445b0 2024-03-12T03:28:24.126 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:23.936+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 279 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:24.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:23.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:24.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:23.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:24.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:23.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:24.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:23.825+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:24.123+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:24.123+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:24.123+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:24.124+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:24.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:24.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:24.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:24.373+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:25.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:24.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:25.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:24.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:25.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:24.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:25.113 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:24.852+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:25.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:25.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:25.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:25.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:25.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:25.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:25.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:25.111+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:25.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:25.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:25.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:25.384+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:26.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:25.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:26.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:25.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:26.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:25.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:26.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:25.835+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:26.383 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:26.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:26.383 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:26.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:26.383 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:26.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:26.383 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:26.108+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:26.380+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:26.380+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:26.380+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:26.381+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:26.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:26.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:26.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:26.857+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:27.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:27.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:27.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:27.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:27.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:27.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:27.077+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:27.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:27.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:27.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:27.355+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:27.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:27.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:27.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:27.879+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:28.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:28.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:28.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:28.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:28.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:28.057+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:28.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:28.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:28.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:28.349+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80022f0 tx=0x7fb7f004cd10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80022f0 tx=0x7fb7f004cd10 comp rx=0 tx=0).stop 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 msgr2=0x7fb7e8044e30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80445b0 0x7fb7e8044e30 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c190 con 0x7fb7e8049c50 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045370 con 0x7fb8000ffcc0 2024-03-12T03:28:28.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:28.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:28.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:28.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80454b0 con 0x7fb7e8049c50 2024-03-12T03:28:28.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d360 tx=0x7fb7f800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:28.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e8049c50 2024-03-12T03:28:28.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000d1c0 con 0x7fb7e8049c50 2024-03-12T03:28:28.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:28.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8049c50 2024-03-12T03:28:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:28.937+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 296 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:28.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:28.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:28.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:28.897+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:29.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:29.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:29.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:29.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:29.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:29.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:29.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:29.078+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:29.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:29.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:29.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:29.314+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:30.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:30.047+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:30.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:30.047+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:30.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:30.047+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:30.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:30.049+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:30.308 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:29.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:30.308 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:29.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:30.308 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:29.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:30.308 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:29.899+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:30.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:30.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:30.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:30.305+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:30.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:30.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:30.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:30.894+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:31.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:31.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:31.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:31.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:31.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:31.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:31.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:31.049+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:31.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:31.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:31.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:31.314+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:32.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:31.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:32.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:31.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:32.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:31.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:32.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:31.851+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:32.282 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:32.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:32.282 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:32.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:32.282 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:32.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:32.282 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:32.014+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:32.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:32.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:32.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:32.279+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:32.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:32.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:32.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:32.869+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:33.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:33.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:33.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:33.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:33.035+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:33.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:33.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:33.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:33.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:33.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:28:33.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:33.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d360 tx=0x7fb7f800c950 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:33.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d360 tx=0x7fb7f800c950 comp rx=0 tx=0).stop 2024-03-12T03:28:33.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:33.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:33.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 0x7fb7e8045210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:33.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:33.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cd40 con 0x7fb7e8044e30 2024-03-12T03:28:33.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c190 con 0x7fb8000ffcc0 2024-03-12T03:28:33.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.938+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 0x7fb7e8045210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:33.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.938+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:33.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.938+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:33.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.938+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045750 con 0x7fb7e8044e30 2024-03-12T03:28:33.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 0x7fb7e8045210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8049680 tx=0x7fb7f0009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:33.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e8044e30 2024-03-12T03:28:33.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0043970 con 0x7fb7e8044e30 2024-03-12T03:28:33.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:33.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8044e30 2024-03-12T03:28:34.286 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:33.937+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 314 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:34.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:34.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:34.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:34.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:34.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:34.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:34.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:34.069+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:34.287 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:33.859+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:34.287 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:33.859+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:34.288 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:33.859+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:34.288 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:33.859+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:34.283+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:34.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:34.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:34.285+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:35.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:35.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:35.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:35.060+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:34.889+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:34.889+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:35.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:34.889+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:35.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:34.890+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:35.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:35.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:35.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:35.320+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:35.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:35.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:35.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:35.902+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:36.042+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:36.042+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:36.042+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:36.043+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:36.342+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:36.342+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:36.342+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:36.343+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:37.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:37.239+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:28:37.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:37.239+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:37.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:37.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:37.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:37.039+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:36.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:36.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:36.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:37.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:36.916+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:37.315+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:37.315+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:37.315+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:37.317+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:37.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:37.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:37.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:37.935+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:38.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:38.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:38.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:38.011+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:38.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:38.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:38.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:38.338+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 msgr2=0x7fb7e8045210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 msgr2=0x7fb7e8045210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 0x7fb7e8045210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8049680 tx=0x7fb7f0009120 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 0x7fb7e8045210 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8049680 tx=0x7fb7f0009120 comp rx=0 tx=0).stop 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 msgr2=0x7fb7e8045210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e30 0x7fb7e8045210 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:38.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.937+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:38.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045750 con 0x7fb7e8049c50 2024-03-12T03:28:38.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045890 con 0x7fb8000ffcc0 2024-03-12T03:28:38.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.938+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:38.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.939+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:38.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:38.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.939+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80459d0 con 0x7fb7e8049c50 2024-03-12T03:28:38.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003a610 tx=0x7fb7f000b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:38.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e8049c50 2024-03-12T03:28:38.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb00 con 0x7fb7e8049c50 2024-03-12T03:28:38.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:38.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8049c50 2024-03-12T03:28:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:38.937+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 330 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:38.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:38.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:38.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:38.932+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:39.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:39.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:39.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:39.027+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:39.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:39.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:39.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:39.362+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:39.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:39.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:39.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:39.961+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:40.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:40.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:40.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:40.062+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:40.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:40.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:40.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:40.338+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:41.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:41.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:41.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:41.033+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:40.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:40.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:40.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:40.927+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:41.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:41.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:41.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:41.374+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:42.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:42.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:42.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:42.076+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:41.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:41.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:41.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:41.878+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:42.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:42.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:42.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:42.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:42.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:42.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:42.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:42.401+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:42.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:42.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:42.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:42.855+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:43.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:43.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:43.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:43.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:43.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:43.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:43.386 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:43.120+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:43.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:43.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:43.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:43.447+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.937+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003a610 tx=0x7fb7f000b1b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003a610 tx=0x7fb7f000b1b0 comp rx=0 tx=0).stop 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 0x7fb7e80451f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cd20 con 0x7fb7e8044e10 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045730 con 0x7fb8000ffcc0 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 0x7fb7e80451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:43.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:43.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.938+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045870 con 0x7fb7e8044e10 2024-03-12T03:28:43.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.939+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 0x7fb7e80451f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017de0 tx=0x7fb7f00529e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:43.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e8044e10 2024-03-12T03:28:43.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8049740 con 0x7fb7e8044e10 2024-03-12T03:28:43.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:43.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800db40 con 0x7fb7e8044e10 2024-03-12T03:28:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:43.938+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 347 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:44.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:44.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:44.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:44.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:44.119+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:43.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:43.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:43.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:43.901+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:44.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:44.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:44.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:44.478+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:45.219 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:44.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:45.219 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:44.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:45.219 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:44.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:45.219 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:44.891+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:45.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:45.134+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:45.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:45.134+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:45.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:45.134+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:45.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:45.135+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:45.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:45.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:45.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:45.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:45.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:45.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:45.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:45.929+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:46.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:46.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:46.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:46.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:46.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:46.172+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:46.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:46.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:46.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:46.531+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:47.210 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:46.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:47.210 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:46.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:47.210 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:46.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:47.210 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:46.939+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:47.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:47.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:47.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:47.208+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:47.973 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:47.563+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:47.973 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:47.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:47.973 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:47.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:47.973 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:47.564+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:48.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:48.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:48.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:48.216+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:47.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:47.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:47.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:47.971+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:48.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 msgr2=0x7fb7e80451f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:28:48.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 msgr2=0x7fb7e80451f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:48.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 0x7fb7e80451f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017de0 tx=0x7fb7f00529e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:48.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 0x7fb7e80451f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017de0 tx=0x7fb7f00529e0 comp rx=0 tx=0).stop 2024-03-12T03:28:48.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 msgr2=0x7fb7e80451f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:48.942 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:48.559+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:48.942 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:48.559+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:48.942 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:48.559+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:48.942 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:48.560+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:48.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8044e10 0x7fb7e80451f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:48.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cd20 con 0x7fb7e8049c50 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045730 con 0x7fb8000ffcc0 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.939+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.939+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045870 con 0x7fb7e8049c50 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.940+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b550 tx=0x7fb7f8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:48.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e8049c50 2024-03-12T03:28:48.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb00 con 0x7fb7e8049c50 2024-03-12T03:28:48.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:48.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e8049c50 2024-03-12T03:28:49.252 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:48.938+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 364 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:49.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:48.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:49.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:48.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:49.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:48.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:49.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:48.987+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:49.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:49.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:49.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:49.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:49.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:49.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:49.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:49.250+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:50.007 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:49.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:50.008 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:49.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:50.008 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:49.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:50.008 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:49.588+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:50.272 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:50.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:50.272 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:50.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:50.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:50.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:50.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:50.005+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:50.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:50.202+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:50.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:50.202+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:50.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:50.202+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:50.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:50.202+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:50.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:50.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:50.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:50.544+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:51.185+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:51.185+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:51.185+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:51.186+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.028+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:51.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:51.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:51.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:51.505+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:52.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:52.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:52.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:52.196+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:51.993+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:52.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:52.599+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:28:52.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:52.599+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:28:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:52.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:52.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:52.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:52.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:53.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:52.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:53.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:52.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:53.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:52.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:53.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:52.971+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:53.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:53.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:53.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:53.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:53.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:53.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:53.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:53.230+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:53.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:53.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:53.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:53.522+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:53.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:28:53.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:53.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b550 tx=0x7fb7f8000f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:53.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b550 tx=0x7fb7f8000f40 comp rx=0 tx=0).stop 2024-03-12T03:28:53.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:53.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:53.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:53.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:53.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cd20 con 0x7fb7e80451f0 2024-03-12T03:28:53.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.938+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045b10 con 0x7fb8000ffcc0 2024-03-12T03:28:53.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:53.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.940+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:53.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.940+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:53.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.940+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045c50 con 0x7fb7e80451f0 2024-03-12T03:28:53.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.940+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003fd0 tx=0x7fb7f003a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:53.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e80451f0 2024-03-12T03:28:53.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb00 con 0x7fb7e80451f0 2024-03-12T03:28:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:53.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e80451f0 2024-03-12T03:28:54.221 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:53.938+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 382 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.014+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:54.476 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:54.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:54.476 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:54.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:54.476 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:54.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:54.477 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:54.218+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:54.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:54.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:54.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:54.474+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:55.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:55.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:55.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:55.227+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:54.978+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:55.673 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:55.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:55.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:55.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:55.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:55.518+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:55.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:55.519+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:55.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:55.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:55.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:55.979+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:56.224+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:56.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:56.224+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:56.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:56.224+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:56.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:56.225+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:56.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:56.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:56.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:56.554+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:57.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:57.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:57.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:57.009+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:57.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:57.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:57.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:57.220+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:57.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:57.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:57.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:57.531+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:58.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:58.058+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:58.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:58.058+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:58.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:58.058+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:58.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:58.059+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:58.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:58.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:58.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:58.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:58.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:58.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:58.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:58.185+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:58.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:58.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:58.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:58.535+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:28:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:28:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:28:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003fd0 tx=0x7fb7f003a160 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:28:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003fd0 tx=0x7fb7f003a160 comp rx=0 tx=0).stop 2024-03-12T03:28:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:28:58.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:28:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045b10 con 0x7fb7e8049c50 2024-03-12T03:28:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045c50 con 0x7fb8000ffcc0 2024-03-12T03:28:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.939+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:28:58.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.940+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:28:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.940+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:28:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045d90 con 0x7fb7e8049c50 2024-03-12T03:28:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0017ed0 tx=0x7fb7f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:28:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804a470 con 0x7fb7e8049c50 2024-03-12T03:28:58.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f804ce40 con 0x7fb7e8049c50 2024-03-12T03:28:58.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:28:58.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f801db00 con 0x7fb7e8049c50 2024-03-12T03:28:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:28:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:28:58.939+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 398 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:28:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:59.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:59.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:59.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:28:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:28:59.087+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:59.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:59.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:59.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:28:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:28:59.139+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:28:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:59.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:59.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:59.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:28:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:28:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:28:59.532+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:00.394 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:00.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:00.394 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:00.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:00.394 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:00.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:00.394 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:00.099+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:00.394 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:00.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:00.394 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:00.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:00.394 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:00.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:00.395 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:00.160+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:00.514+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:00.514+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:00.514+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:00.515+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:01.175+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:01.175+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:01.175+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:01.175+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:01.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:01.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:01.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:01.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:01.119+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:01.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:01.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:01.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:01.479+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:02.479 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:02.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:02.480 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:02.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:02.480 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:02.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:02.480 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:02.181+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:02.480 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:02.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:02.480 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:02.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:02.480 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:02.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:02.480 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:02.137+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:02.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:02.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:02.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:02.478+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:03.458 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:03.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:03.458 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:03.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:03.458 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:03.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:03.459 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:03.130+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:03.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:03.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:03.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:03.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:03.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:03.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:03.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:03.197+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:03.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:03.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:03.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:03.456+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:03.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0017ed0 tx=0x7fb7f0008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0017ed0 tx=0x7fb7f0008040 comp rx=0 tx=0).stop 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045b10 con 0x7fb7e80451f0 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.939+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045c50 con 0x7fb8000ffcc0 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.940+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.940+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:03.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.940+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:03.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.940+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045d90 con 0x7fb7e80451f0 2024-03-12T03:29:03.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.941+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb7e801a810 tx=0x7fb7f8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:03.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e80451f0 2024-03-12T03:29:03.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e80451f0 2024-03-12T03:29:03.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:03.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e80451f0 2024-03-12T03:29:04.213 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:03.939+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 415 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:04.214 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:04.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:04.214 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:04.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:04.214 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:04.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:04.214 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:04.096+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:04.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:04.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:04.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:04.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:04.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:04.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:04.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:04.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:04.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:04.211+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:05.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:05.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:05.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:05.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:05.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:05.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:05.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:05.104+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:05.455 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:05.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:05.456 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:05.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:05.456 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:05.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:05.456 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:05.413+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:05.456 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:05.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:05.456 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:05.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:05.456 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:05.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:05.456 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:05.192+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:06.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:06.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:06.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:06.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:06.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:06.164+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:06.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:06.165+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:06.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:06.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:06.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:06.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:06.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:06.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:06.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:06.113+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:06.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:06.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:06.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:06.395+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:07.404 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:07.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:07.405 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:07.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:07.405 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:07.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:07.405 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:07.083+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:07.405 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:07.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:07.405 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:07.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:07.405 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:07.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:07.405 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:07.120+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:07.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:07.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:07.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:07.403+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:07.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:07.959+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:29:07.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:07.959+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:29:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:08.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:08.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:08.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:08.098+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:08.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:08.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:08.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:08.154+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:08.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:08.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:08.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:08.377+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:08.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.939+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:29:08.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb7e801a810 tx=0x7fb7f8000f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fb7e801a810 tx=0x7fb7f8000f40 comp rx=0 tx=0).stop 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045b10 con 0x7fb7e8049c50 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045c50 con 0x7fb8000ffcc0 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.940+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:08.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:08.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:08.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045d90 con 0x7fb7e8049c50 2024-03-12T03:29:08.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.942+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8008000 tx=0x7fb7f004bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:08.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804a470 con 0x7fb7e8049c50 2024-03-12T03:29:08.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e8049c50 2024-03-12T03:29:08.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:08.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80493d0 con 0x7fb7e8049c50 2024-03-12T03:29:09.200 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:08.940+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 432 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:09.201 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:09.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:09.201 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:09.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:09.201 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:09.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:09.201 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:09.065+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:09.330+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:09.330+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:09.330+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:09.330+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:09.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:09.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:09.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:09.199+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.036+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:10.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:10.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:10.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:10.218+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:10.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:10.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:10.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:10.349+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:10.988+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:11.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:11.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:11.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:11.196+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:11.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:11.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:11.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:11.362+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:11.945+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:11.945+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:11.945+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:11.945+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:12.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:12.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:12.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:12.181+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:12.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:12.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:12.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:12.383+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:13.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:12.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:13.230 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:12.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:13.230 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:12.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:13.230 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:12.929+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:13.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:13.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:13.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:13.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:13.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:13.226+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:13.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:13.227+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:13.530 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:13.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:13.530 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:13.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:13.530 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:13.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:13.530 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:13.371+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:13.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:29:13.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8008000 tx=0x7fb7f004bb20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8008000 tx=0x7fb7f004bb20 comp rx=0 tx=0).stop 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8049c50 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045b10 con 0x7fb7e80451f0 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.940+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8045c50 con 0x7fb8000ffcc0 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:13.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:13.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8045d90 con 0x7fb7e80451f0 2024-03-12T03:29:13.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8045e60 tx=0x7fb7f80094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:13.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804a470 con 0x7fb7e80451f0 2024-03-12T03:29:13.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e80451f0 2024-03-12T03:29:13.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:13.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8031430 con 0x7fb7e80451f0 2024-03-12T03:29:13.946 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:13.940+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 450 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:14.243 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:13.966+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:14.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:13.966+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:14.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:13.966+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:14.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:13.967+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:14.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:14.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:14.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:14.242+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:14.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:14.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:14.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:14.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:14.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:14.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:14.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:14.359+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:15.238 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:14.967+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:15.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:14.967+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:15.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:14.967+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:15.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:14.968+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:15.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:15.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:15.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:15.237+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:15.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:15.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:15.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:15.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:16.190 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:15.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:16.190 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:15.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:16.190 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:15.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:16.191 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:15.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:16.440 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:16.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:16.441 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:16.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:16.441 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:16.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:16.441 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:16.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:16.441 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:16.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:16.441 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:16.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:16.441 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:16.188+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:16.438+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:17.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:17.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:17.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:17.193+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:16.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:16.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:16.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:16.962+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:17.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:17.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:17.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:17.400+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:18.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:18.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:18.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:18.206+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:17.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:17.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:17.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:17.988+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:18.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:18.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:18.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:18.434+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8045e60 tx=0x7fb7f80094a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8045e60 tx=0x7fb7f80094a0 comp rx=0 tx=0).stop 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 msgr2=0x7fb7e80455d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80451f0 0x7fb7e80455d0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 0x7fb7e804d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804f760 con 0x7fb7e8045b10 2024-03-12T03:29:18.944 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804f8a0 con 0x7fb8000ffcc0 2024-03-12T03:29:18.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 0x7fb7e804d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:18.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.942+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:18.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.942+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:18.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.942+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804f9e0 con 0x7fb7e8045b10 2024-03-12T03:29:18.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.942+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 0x7fb7e804d210 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804def0 tx=0x7fb7f80490d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:18.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801d330 con 0x7fb7e8045b10 2024-03-12T03:29:18.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801d330 con 0x7fb7e8045b10 2024-03-12T03:29:18.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:18.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8031430 con 0x7fb7e8045b10 2024-03-12T03:29:19.243 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:18.941+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 466 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:19.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:19.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:19.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:19.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:19.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:19.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:19.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:19.033+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:19.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:19.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:19.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:19.241+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:19.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:19.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:19.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:19.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:20.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:20.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:20.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:20.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.013+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:20.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:20.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:20.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:20.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:20.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:20.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:20.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:20.403+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:21.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.991+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:21.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.991+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:21.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.991+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:21.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:20.992+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:21.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:21.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:21.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:21.300+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:21.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:21.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:21.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:21.450+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:22.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:21.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:22.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:21.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:22.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:21.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:22.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:21.948+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:22.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:22.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:22.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:22.467+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:22.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:22.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:22.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:22.276+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:23.238 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:22.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:23.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:22.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:23.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:22.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:23.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:22.935+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:23.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.319+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:29:23.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.319+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:29:23.499 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:23.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:23.499 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:23.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:23.499 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:23.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:23.499 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:23.234+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:23.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:23.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:23.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:23.497+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 msgr2=0x7fb7e804d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 0x7fb7e804d210 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804def0 tx=0x7fb7f80490d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 0x7fb7e804d210 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804def0 tx=0x7fb7f80490d0 comp rx=0 tx=0).stop 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 msgr2=0x7fb7e804d210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e8045b10 0x7fb7e804d210 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:23.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 0x7fb7e804f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:23.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80459b0 con 0x7fb7e80455d0 2024-03-12T03:29:23.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.941+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fca0 con 0x7fb8000ffcc0 2024-03-12T03:29:23.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.942+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 0x7fb7e804f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:23.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.942+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:23.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.942+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:23.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.943+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804fde0 con 0x7fb7e80455d0 2024-03-12T03:29:23.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.943+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 0x7fb7e804f760 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:23.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801d330 con 0x7fb7e80455d0 2024-03-12T03:29:23.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8031730 con 0x7fb7e80455d0 2024-03-12T03:29:23.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:23.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8031430 con 0x7fb7e80455d0 2024-03-12T03:29:24.271 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:23.941+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 483 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:24.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:23.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:24.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:23.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:24.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:23.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:24.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:23.937+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:24.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:24.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:24.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:24.504+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:24.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:24.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:24.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:24.269+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:25.275 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:24.893+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:25.275 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:24.893+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:25.275 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:24.893+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:25.275 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:24.893+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:25.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:25.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:25.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:25.273+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:25.513+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:25.513+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:25.513+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:25.514+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:25.910+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:25.910+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:25.910+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:25.910+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:26.283+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:26.283+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:26.283+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:26.284+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:26.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:26.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:26.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:26.474+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:27.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:26.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:27.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:26.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:27.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:26.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:27.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:26.928+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:27.287+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:27.287+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:27.287+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:27.288+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:27.427+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:27.427+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:27.427+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:27.428+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:28.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:27.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:28.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:27.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:28.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:27.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:28.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:27.941+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:28.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:28.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:28.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:28.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:28.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:28.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:28.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:28.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:28.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:28.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:28.460+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 msgr2=0x7fb7e804f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.941+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 msgr2=0x7fb7e804f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 0x7fb7e804f760 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 0x7fb7e804f760 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 msgr2=0x7fb7e804f760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e80455d0 0x7fb7e804f760 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80459b0 con 0x7fb7e804d210 2024-03-12T03:29:28.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804d5f0 con 0x7fb8000ffcc0 2024-03-12T03:29:28.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.942+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:28.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.943+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:28.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.943+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:28.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.943+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80501e0 con 0x7fb7e804d210 2024-03-12T03:29:28.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.943+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804fca0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8029410 tx=0x7fb7f8045e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:28.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804d210 2024-03-12T03:29:28.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d210 2024-03-12T03:29:28.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:28.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0059040 con 0x7fb7e804d210 2024-03-12T03:29:29.287 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:28.942+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 500 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:29.287 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:28.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:29.287 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:28.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:29.287 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:28.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:29.287 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:28.955+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:29.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:29.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:29.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:29.416+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:29.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:29.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:29.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:29.285+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:30.246 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:29.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:30.246 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:29.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:30.247 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:29.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:30.247 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:29.926+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:30.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:30.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:30.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:30.244+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:30.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:30.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:30.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:30.377+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:30.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:30.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:30.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:30.902+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:31.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:31.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:31.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:31.370+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:31.202+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:31.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:31.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:31.204+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:31.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:31.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:31.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:31.892+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:32.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:32.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:32.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:32.414+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:32.209+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:32.209+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:32.209+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:32.210+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:33.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:32.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:33.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:32.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:33.263 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:32.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:33.263 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:32.856+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:33.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:33.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:33.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:33.260+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:33.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:33.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:33.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:33.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:33.366+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:33.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:29:33.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:33.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804fca0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8029410 tx=0x7fb7f8045e90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804fca0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8029410 tx=0x7fb7f8045e90 comp rx=0 tx=0).stop 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804fca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804fca0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e80501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80459b0 con 0x7fb7e804d5f0 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804f9e0 con 0x7fb8000ffcc0 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.943+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e80501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.943+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.943+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:33.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.943+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050720 con 0x7fb7e804d5f0 2024-03-12T03:29:33.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.943+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e80501e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:33.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d5f0 2024-03-12T03:29:33.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b8c0 con 0x7fb7e804d5f0 2024-03-12T03:29:33.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:33.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000f6d0 con 0x7fb7e804d5f0 2024-03-12T03:29:34.225 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:33.942+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 518 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:34.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:33.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:34.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:33.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:34.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:33.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:34.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:33.830+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:34.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:34.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:34.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:34.223+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:34.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:34.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:34.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:34.416+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:35.203 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:34.818+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:35.203 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:34.818+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:35.203 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:34.818+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:35.203 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:34.819+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:35.200+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:35.200+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:35.200+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:35.201+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:35.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:35.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:35.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:35.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:35.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:35.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:35.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:35.416+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:35.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:35.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:35.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:35.811+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:36.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:36.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:36.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:36.199+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:36.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:36.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:36.412+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:36.413+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:36.806+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:36.806+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:36.806+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:36.807+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:37.151+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:37.151+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:37.151+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:37.152+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:37.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:37.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:37.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:37.389+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:37.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:37.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:37.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:37.804+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:38.377 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:38.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:38.377 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:38.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:38.377 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:38.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:38.377 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:38.116+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:38.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.679+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:29:38.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.679+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:29:38.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:38.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:38.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:38.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:38.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:38.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:38.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:38.375+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:38.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e80501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:29:38.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e80501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:38.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e80501e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f004c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:38.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e80501e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f004c000 comp rx=0 tx=0).stop 2024-03-12T03:29:38.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e80501e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:38.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e80501e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:38.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:38.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:38.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804ff20 con 0x7fb7e804d210 2024-03-12T03:29:38.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.942+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050060 con 0x7fb8000ffcc0 2024-03-12T03:29:38.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:38.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:38.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:38.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:38.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:38.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:38.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:38.841+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:38.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.943+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:38.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:38.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:38.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050720 con 0x7fb7e804d210 2024-03-12T03:29:38.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802eee0 tx=0x7fb7f004cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:38.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8031a10 con 0x7fb7e804d210 2024-03-12T03:29:38.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8031a10 con 0x7fb7e804d210 2024-03-12T03:29:38.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:38.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8031430 con 0x7fb7e804d210 2024-03-12T03:29:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:38.942+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 534 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:39.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:39.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:39.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:39.090+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:39.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:39.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:39.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:39.405+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:39.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:39.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:39.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:39.823+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:40.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:40.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:40.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:40.060+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:40.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:40.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:40.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:40.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:40.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:40.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:40.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:40.415+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:41.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:41.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:41.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:41.054+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:40.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:40.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:41.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:40.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:41.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:40.806+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:41.754 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:41.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:41.754 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:41.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:41.755 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:41.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:41.755 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:41.423+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:42.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:42.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:42.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:42.053+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:41.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:41.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:41.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:41.840+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:42.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:42.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:42.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:42.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:42.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:42.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:42.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:42.408+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:42.794+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:42.794+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:42.794+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:42.795+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:43.443 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:43.067+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:43.443 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:43.067+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:43.443 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:43.067+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:43.443 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:43.067+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:43.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:43.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:43.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:43.441+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:43.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:29:43.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:43.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802eee0 tx=0x7fb7f004cd10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802eee0 tx=0x7fb7f004cd10 comp rx=0 tx=0).stop 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb7e804d5f0 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80506e0 con 0x7fb8000ffcc0 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.943+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050820 con 0x7fb7e804d5f0 2024-03-12T03:29:43.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f8008850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:43.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8050aa0 con 0x7fb7e804d5f0 2024-03-12T03:29:43.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8050aa0 con 0x7fb7e804d5f0 2024-03-12T03:29:43.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:43.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8031430 con 0x7fb7e804d5f0 2024-03-12T03:29:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:43.943+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 551 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:44.073+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:44.073+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:44.073+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:44.074+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:43.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:43.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:43.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:43.845+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:44.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:44.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:44.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:44.475+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:45.118 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:44.860+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:45.118 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:44.860+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:45.118 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:44.860+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:45.118 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:44.861+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:45.432 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:45.115+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:45.432 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:45.115+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:45.432 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:45.115+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:45.432 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:45.116+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:45.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:45.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:45.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:45.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:46.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:45.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:46.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:45.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:46.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:45.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:46.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:45.859+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:46.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:46.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:46.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:46.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:46.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:46.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:46.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:46.119+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:46.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:46.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:46.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:46.386+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:46.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:46.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:46.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:46.846+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:47.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:47.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:47.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:47.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:47.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:47.091+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:47.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:47.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:47.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:47.369+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:48.119 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:47.865+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:48.120 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:47.865+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:48.120 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:47.865+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:48.120 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:47.866+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:48.411 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:48.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:48.411 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:48.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:48.412 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:48.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:48.412 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:48.117+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:48.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:48.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:48.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:48.409+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:48.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:29:48.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f8008850 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017be0 tx=0x7fb7f8008850 comp rx=0 tx=0).stop 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050510 con 0x7fb7e804d210 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.943+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050650 con 0x7fb8000ffcc0 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.944+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.944+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.944+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:48.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.944+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050790 con 0x7fb7e804d210 2024-03-12T03:29:48.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.945+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b550 tx=0x7fb7f8030d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:48.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d210 2024-03-12T03:29:48.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804d210 2024-03-12T03:29:48.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:48.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000f6d0 con 0x7fb7e804d210 2024-03-12T03:29:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:48.943+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 568 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:49.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:49.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:49.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:49.095+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:48.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:48.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:48.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:48.853+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:49.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:49.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:49.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:49.449+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:50.090 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:49.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:50.090 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:49.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:50.090 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:49.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:50.091 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:49.830+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:50.403 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:50.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:50.403 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:50.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:50.403 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:50.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:50.403 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:50.089+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:50.403 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:50.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:50.403 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:50.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:50.403 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:50.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:50.401+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:51.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:51.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:51.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:51.062+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:50.879+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:50.879+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:50.879+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:50.880+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:51.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:51.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:51.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:51.397+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:52.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:52.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:52.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:52.039+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:51.895+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:51.895+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:51.895+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:51.896+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:52.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:52.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:52.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:52.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:52.382+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:53.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:53.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:53.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:53.013+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:52.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:53.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:52.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:53.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:52.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:53.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:52.929+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:53.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:53.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:53.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:53.362+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:53.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b550 tx=0x7fb7f8030d20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b550 tx=0x7fb7f8030d20 comp rx=0 tx=0).stop 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050460 con 0x7fb7e804d5f0 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb8000ffcc0 2024-03-12T03:29:53.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:53.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:53.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:53.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80506e0 con 0x7fb7e804d5f0 2024-03-12T03:29:53.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.945+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804e280 tx=0x7fb7f0044a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:53.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801dcd0 con 0x7fb7e804d5f0 2024-03-12T03:29:53.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801db00 con 0x7fb7e804d5f0 2024-03-12T03:29:53.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:53.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8030d90 con 0x7fb7e804d5f0 2024-03-12T03:29:54.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:54.039+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:29:54.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:54.039+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:29:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:53.944+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 595 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:54.057+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:54.057+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:54.057+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:54.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:54.058+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:53.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:53.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:53.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:53.969+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:54.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:54.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:54.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:54.387+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:55.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:55.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:55.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:55.020+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:55.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:55.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:55.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:55.013+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:55.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:55.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:55.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:55.370+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:56.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:56.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:56.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:56.051+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:56.038+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:56.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:56.038+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:56.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:56.038+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:56.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:56.038+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:56.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:56.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:56.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:56.355+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:57.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:57.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:57.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:57.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:57.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:57.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:57.062+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:57.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:57.048+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:57.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:57.048+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:57.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:57.048+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:57.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:57.048+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:57.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:57.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:57.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:57.341+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:58.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:58.351 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:58.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:58.351 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:58.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:58.351 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:58.096+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:58.352 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:58.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:58.352 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:58.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:58.352 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:58.035+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:58.352 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:58.036+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:58.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:58.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:58.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:58.349+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:29:58.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:29:58.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:29:58.947 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804e280 tx=0x7fb7f0044a20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804e280 tx=0x7fb7f0044a20 comp rx=0 tx=0).stop 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.944+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050460 con 0x7fb7e804d210 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb8000ffcc0 2024-03-12T03:29:58.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.945+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:29:58.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:29:58.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:29:58.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80506e0 con 0x7fb7e804d210 2024-03-12T03:29:58.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003c010 tx=0x7fb7f8027fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:29:58.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d210 2024-03-12T03:29:58.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b870 con 0x7fb7e804d210 2024-03-12T03:29:58.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:29:58.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0053570 con 0x7fb7e804d210 2024-03-12T03:29:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:29:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:29:58.944+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 641 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:29:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:59.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:59.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:59.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:29:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:29:59.083+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:59.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:59.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:59.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:29:59.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.008+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:29:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:59.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:59.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:59.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:29:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:29:59.395+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:29:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:29:59.971+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:00.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:00.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:00.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:00.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:00.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:00.069+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:00.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:00.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:00.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:00.422+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:01.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:01.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:01.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:01.041+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:00.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:00.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:01.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:00.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:01.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:00.971+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:01.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:01.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:01.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:01.453+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:01.957+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:01.957+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:01.957+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:01.957+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:02.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:02.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:02.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:02.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:02.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:02.077+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:02.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:02.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:02.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:02.410+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:03.067+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:03.067+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:03.067+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:03.067+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:02.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:02.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:02.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:02.937+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:03.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:03.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:03.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:03.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:03.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:03.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:03.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:03.382+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:03.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:03.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:03.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003c010 tx=0x7fb7f8027fd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:03.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003c010 tx=0x7fb7f8027fd0 comp rx=0 tx=0).stop 2024-03-12T03:30:03.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80504f0 con 0x7fb7e804d5f0 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050630 con 0x7fb8000ffcc0 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.945+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050770 con 0x7fb7e804d5f0 2024-03-12T03:30:03.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004c490 tx=0x7fb7f00090e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:03.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804d5f0 2024-03-12T03:30:03.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a760 con 0x7fb7e804d5f0 2024-03-12T03:30:03.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:03.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0053570 con 0x7fb7e804d5f0 2024-03-12T03:30:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:03.945+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 706 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:04.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:04.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:04.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:04.103+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:03.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:03.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:03.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:03.953+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:04.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:04.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:04.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:04.389+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:05.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:05.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:05.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:05.103+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:04.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:04.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:04.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:04.907+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:05.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:05.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:05.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:05.384+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:05.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:05.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:05.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:05.955+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:06.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:06.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:06.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:06.059+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:06.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:06.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:06.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:06.369+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:06.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:06.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:06.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:06.949+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:07.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:07.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:07.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:07.036+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:07.688 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:07.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:07.689 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:07.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:07.689 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:07.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:07.689 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:07.397+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:08.079+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:08.079+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:08.079+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:08.079+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:07.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:07.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:07.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:07.976+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:08.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:08.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:08.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:08.411+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004c490 tx=0x7fb7f00090e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004c490 tx=0x7fb7f00090e0 comp rx=0 tx=0).stop 2024-03-12T03:30:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:08.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:08.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e804d210 2024-03-12T03:30:08.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80504f0 con 0x7fb8000ffcc0 2024-03-12T03:30:08.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.945+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:08.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.946+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:08.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.946+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:08.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.946+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050630 con 0x7fb7e804d210 2024-03-12T03:30:08.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.946+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804cd60 tx=0x7fb7f0034540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803f9f0 con 0x7fb7e804d210 2024-03-12T03:30:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f80317b0 con 0x7fb7e804d210 2024-03-12T03:30:08.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:08.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8030d90 con 0x7fb7e804d210 2024-03-12T03:30:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:08.945+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 776 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:09.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:09.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:09.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:09.083+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:09.011+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:09.011+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:09.011+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:09.011+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:09.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:09.463+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:30:09.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:09.463+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:30:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:09.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:09.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:09.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:09.394+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.002+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:10.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:10.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:10.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:10.122+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:10.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:10.355+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:10.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:10.355+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:10.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:10.355+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:10.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:10.356+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:10.960+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:11.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:11.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:11.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:11.162+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:11.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:11.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:11.391+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:11.392+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:11.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:11.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:11.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:11.972+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:12.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:12.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:12.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:12.178+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:12.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:12.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:12.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:12.424+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:13.193 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:12.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:13.193 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:12.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:13.193 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:12.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:13.193 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:12.938+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:13.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:13.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:13.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:13.429+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:13.190+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:13.190+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:13.190+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:13.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:13.191+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:13.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:30:13.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:13.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804cd60 tx=0x7fb7f0034540 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804cd60 tx=0x7fb7f0034540 comp rx=0 tx=0).stop 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050460 con 0x7fb7e804d5f0 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb8000ffcc0 2024-03-12T03:30:13.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.947+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:13.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.948+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:13.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.948+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:13.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.948+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80506e0 con 0x7fb7e804d5f0 2024-03-12T03:30:13.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.948+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0023f90 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:13.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d1c0 con 0x7fb7e804d5f0 2024-03-12T03:30:13.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0045820 con 0x7fb7e804d5f0 2024-03-12T03:30:13.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:13.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000f960 con 0x7fb7e804d5f0 2024-03-12T03:30:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:13.946+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 842 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:14.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:14.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:14.171+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:14.172+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:13.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:13.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:13.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:13.942+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:14.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:14.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:14.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:14.434+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:15.223 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:14.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:15.223 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:14.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:15.223 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:14.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:15.224 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:14.949+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:15.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:15.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:15.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:15.402+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:15.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:15.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:15.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:15.221+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:16.251 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:15.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:16.251 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:15.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:16.251 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:15.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:16.251 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:15.956+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:16.248+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:16.248+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:16.248+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:16.249+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:16.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:16.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:16.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:16.418+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:17.263 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:16.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:17.263 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:16.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:17.263 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:16.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:17.263 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:16.981+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:17.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:17.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:17.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:17.261+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:17.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:17.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:17.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:17.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:18.023+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:18.023+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:18.023+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:18.024+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:18.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:18.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:18.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:18.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:18.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:18.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:18.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:18.242+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:18.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:18.840 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:18.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:18.840 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:18.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:18.840 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:18.432+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:18.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:30:18.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:18.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0023f90 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0023f90 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050460 con 0x7fb7e804d210 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb8000ffcc0 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.947+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.947+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:18.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.947+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80506e0 con 0x7fb7e804d210 2024-03-12T03:30:18.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.947+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b9c0 tx=0x7fb7f000b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:18.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d1c0 con 0x7fb7e804d210 2024-03-12T03:30:18.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00336e0 con 0x7fb7e804d210 2024-03-12T03:30:18.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:18.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000dcc0 con 0x7fb7e804d210 2024-03-12T03:30:19.233 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:18.946+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 903 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:19.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:19.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:19.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:19.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:19.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:19.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:19.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:19.040+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:19.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:19.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:19.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:19.232+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:19.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:19.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:19.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:19.442+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:20.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:20.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:20.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:20.032+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:20.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:20.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:20.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:20.260+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:20.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:20.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:20.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:20.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:20.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:20.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:20.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:20.448+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:21.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:21.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:21.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:21.019+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:21.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:21.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:21.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:21.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:21.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:21.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:21.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:21.260+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:21.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:21.442+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:21.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:21.442+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:21.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:21.442+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:21.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:21.443+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:22.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:22.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:22.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:22.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:22.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:22.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:22.302 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:22.018+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:22.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:22.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:22.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:22.300+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:22.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:22.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:22.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:22.486+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:23.508 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:23.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:23.508 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:23.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:23.508 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:23.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:23.508 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:23.276+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:23.508 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:23.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:23.509 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:23.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:23.509 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:23.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:23.509 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:23.062+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:23.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:23.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:23.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:23.506+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:23.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b9c0 tx=0x7fb7f000b1b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b9c0 tx=0x7fb7f000b1b0 comp rx=0 tx=0).stop 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050460 con 0x7fb7e804d5f0 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.946+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb8000ffcc0 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.947+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.947+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:23.950 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.947+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:23.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.947+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80506e0 con 0x7fb7e804d5f0 2024-03-12T03:30:23.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.948+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0058ef0 tx=0x7fb7f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:23.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d5f0 2024-03-12T03:30:23.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00336e0 con 0x7fb7e804d5f0 2024-03-12T03:30:23.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:23.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000dcc0 con 0x7fb7e804d5f0 2024-03-12T03:30:24.267 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:23.947+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 966 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:24.267 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:24.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:24.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:24.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:24.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:24.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:24.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:24.100+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:24.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:24.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:24.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:24.265+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:24.483+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:24.483+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:24.483+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:24.484+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:24.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:24.823+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:30:24.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:24.823+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:30:25.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:25.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:25.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:25.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:25.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:25.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:25.520 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:25.132+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:25.520 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:25.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:25.520 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:25.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:25.520 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:25.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:25.520 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:25.230+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:25.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:25.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:25.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:25.517+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:26.371 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:26.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:26.372 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:26.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:26.372 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:26.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:26.372 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:26.234+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:26.372 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:26.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:26.372 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:26.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:26.372 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:26.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:26.372 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:26.089+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:26.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:26.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:26.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:26.566+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:27.094+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:27.094+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:27.094+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:27.095+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:27.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:27.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:27.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:27.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:27.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:27.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:27.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:27.261+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:27.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:27.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:27.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:27.572+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:28.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:28.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:28.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:28.069+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:28.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:28.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:28.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:28.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:28.950 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:28.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:28.950 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:28.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:28.950 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:28.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:28.950 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:28.587+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0058ef0 tx=0x7fb7f000b890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0058ef0 tx=0x7fb7f000b890 comp rx=0 tx=0).stop 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050460 con 0x7fb7e804d210 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb8000ffcc0 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.947+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:28.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.948+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:28.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.948+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:28.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.948+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80506e0 con 0x7fb7e804d210 2024-03-12T03:30:28.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.948+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8047ea0 tx=0x7fb7f000bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:28.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fbc0 con 0x7fb7e804d210 2024-03-12T03:30:28.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803f170 con 0x7fb7e804d210 2024-03-12T03:30:28.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:28.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8030d90 con 0x7fb7e804d210 2024-03-12T03:30:29.299 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:28.948+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1029 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:29.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:29.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:29.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:29.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:29.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:29.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:29.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:29.092+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:29.559 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:29.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:29.559 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:29.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:29.559 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:29.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:29.559 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:29.298+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:29.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:29.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:29.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:29.557+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:30.121+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:30.121+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:30.121+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:30.122+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:30.528 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:30.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:30.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:30.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:30.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:30.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:30.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:30.323+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:30.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:30.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:30.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:30.526+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:31.424 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:31.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:31.424 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:31.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:31.424 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:31.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:31.424 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:31.083+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:31.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:31.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:31.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:31.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:31.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:31.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:31.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:31.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:31.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:31.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:31.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:31.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:32.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:32.286+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:32.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:32.286+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:32.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:32.286+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:32.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:32.286+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:32.536 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:32.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:32.536 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:32.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:32.536 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:32.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:32.536 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:32.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:32.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:32.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:32.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:32.533+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:33.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:33.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:33.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:33.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:33.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:33.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:33.301 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:33.045+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:33.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:33.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:33.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:33.298+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:33.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:33.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:33.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:33.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:33.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:33.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:33.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:33.520+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8047ea0 tx=0x7fb7f000bc90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8047ea0 tx=0x7fb7f000bc90 comp rx=0 tx=0).stop 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:33.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:33.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050460 con 0x7fb7e804d5f0 2024-03-12T03:30:33.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80505a0 con 0x7fb8000ffcc0 2024-03-12T03:30:33.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.948+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:33.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.949+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:33.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.949+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:33.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.949+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80506e0 con 0x7fb7e804d5f0 2024-03-12T03:30:33.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.949+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000ec90 tx=0x7fb7f8015bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:33.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d5f0 2024-03-12T03:30:33.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000ee30 con 0x7fb7e804d5f0 2024-03-12T03:30:33.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:33.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000dcc0 con 0x7fb7e804d5f0 2024-03-12T03:30:34.276 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:33.948+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1094 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:34.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:34.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:34.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:34.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.007+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:34.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:34.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:34.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:34.275+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:34.970 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:34.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:34.970 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:34.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:34.971 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:34.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:34.971 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:34.570+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:35.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:35.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:35.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:35.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:34.968+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:35.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:35.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:35.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:35.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:35.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:35.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:35.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:35.255+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:35.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:35.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:35.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:35.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:36.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:36.273+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e0005870 con 0x7fb7e804d5f0 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:36.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:36.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:36.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:36.237+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:36.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:36.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:36.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:36.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:36.017+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:37.029 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:36.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:37.029 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:36.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:37.029 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:36.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:37.029 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:36.573+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:37.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:37.273+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e00057f0 con 0x7fb7e804d5f0 2024-03-12T03:30:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:37.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:37.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:37.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:37.247+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:37.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:37.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:37.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:37.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:37.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:37.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:37.027+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:37.541+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:37.541+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:37.541+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:37.542+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:38.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.273+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e00057f0 con 0x7fb7e804d5f0 2024-03-12T03:30:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:38.074+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:38.074+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:38.074+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:38.075+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:38.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:38.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:38.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:38.272+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:38.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.948+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:38.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.948+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:38.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.948+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000ec90 tx=0x7fb7f8015bd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:38.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.948+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000ec90 tx=0x7fb7f8015bd0 comp rx=0 tx=0).stop 2024-03-12T03:30:38.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.948+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:38.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.949+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:38.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.949+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:38.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.949+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:38.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050510 con 0x7fb7e804d210 2024-03-12T03:30:38.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050650 con 0x7fb8000ffcc0 2024-03-12T03:30:38.952 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:38.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:38.952 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:38.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:38.952 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:38.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:38.953 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:38.568+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:38.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.949+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:38.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.950+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:38.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.950+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:38.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.950+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050790 con 0x7fb7e804d210 2024-03-12T03:30:38.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.950+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0004510 tx=0x7fb7f8051f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:38.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d210 2024-03-12T03:30:38.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00435b0 con 0x7fb7e804d210 2024-03-12T03:30:38.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:38.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804d210 2024-03-12T03:30:39.283 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:38.949+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1157 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:39.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:39.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:39.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:39.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:39.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:39.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:39.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:39.062+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:39.542 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:39.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:39.542 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:39.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:39.542 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:39.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:39.542 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:39.281+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:39.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:39.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:39.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:39.540+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:40.308 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:40.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:40.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:40.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:40.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:40.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:40.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:40.027+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:40.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:40.311+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:30:40.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:40.311+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:30:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:40.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:40.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:40.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:40.543+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:40.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:40.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:40.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:40.306+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:41.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:41.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:41.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:41.046+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:41.569 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:41.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:41.569 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:41.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:41.569 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:41.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:41.569 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:41.313+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:41.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:41.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:41.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:41.567+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:42.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:42.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:42.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:42.060+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:42.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:42.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:42.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:42.308+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:42.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:42.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:42.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:42.607+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:43.386 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:43.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:43.387 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:43.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:43.387 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:43.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:43.387 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:43.097+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:43.387 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:43.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:43.387 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:43.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:43.387 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:43.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:43.387 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:43.313+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:43.591+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:43.591+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:43.591+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:43.592+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:43.951 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:43.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:43.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0004510 tx=0x7fb7f8051f70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:43.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0004510 tx=0x7fb7f8051f70 comp rx=0 tx=0).stop 2024-03-12T03:30:43.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:43.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:43.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:43.952 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:43.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050510 con 0x7fb7e804d5f0 2024-03-12T03:30:43.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.949+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050650 con 0x7fb8000ffcc0 2024-03-12T03:30:43.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.950+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:43.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.951+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:43.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.951+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:43.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.951+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050790 con 0x7fb7e804d5f0 2024-03-12T03:30:43.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.951+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0043750 tx=0x7fb7f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:43.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f00332f0 con 0x7fb7e804d5f0 2024-03-12T03:30:43.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804d5f0 2024-03-12T03:30:43.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:43.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804d5f0 2024-03-12T03:30:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:43.949+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1221 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:44.128+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:44.128+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:44.128+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:44.128+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:44.569 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:44.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:44.570 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:44.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:44.570 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:44.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:44.570 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:44.337+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:44.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:44.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:44.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:44.567+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:45.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:45.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:45.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:45.118+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:45.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:45.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:45.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:45.314+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:45.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:45.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:45.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:45.586+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:46.459 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:46.125+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:46.459 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:46.125+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:46.459 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:46.125+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:46.459 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:46.126+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:46.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:46.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:46.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:46.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:46.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:46.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:46.460 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:46.347+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:46.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:46.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:46.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:46.606+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:47.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:47.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:47.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:47.354+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:47.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:47.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:47.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:47.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:47.577+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:47.577+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:47.577+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:47.578+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:48.384 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:48.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:48.384 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:48.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:48.384 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:48.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:48.384 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:48.120+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:48.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:48.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:48.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:48.606+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:48.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:48.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:48.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:48.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:48.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:48.382+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.949+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.949+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.949+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0043750 tx=0x7fb7f000b890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.949+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0043750 tx=0x7fb7f000b890 comp rx=0 tx=0).stop 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050650 con 0x7fb7e804d210 2024-03-12T03:30:48.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050790 con 0x7fb8000ffcc0 2024-03-12T03:30:48.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:48.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:48.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:48.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.950+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80508d0 con 0x7fb7e804d210 2024-03-12T03:30:48.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.951+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8021ab0 tx=0x7fb7f0045820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:48.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fbc0 con 0x7fb7e804d210 2024-03-12T03:30:48.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803f170 con 0x7fb7e804d210 2024-03-12T03:30:48.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:48.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8030d90 con 0x7fb7e804d210 2024-03-12T03:30:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:48.949+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1283 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:49.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:49.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:49.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:49.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:49.620 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:49.348+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:49.620 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:49.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:49.620 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:49.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:49.621 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:49.350+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:49.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:49.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:49.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:49.619+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:50.126+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:50.126+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:50.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:50.126+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:50.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:50.127+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:50.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:50.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:50.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:50.368+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:50.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:50.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:50.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:50.660+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:51.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:51.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:51.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:51.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:51.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:51.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:51.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:51.131+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:51.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:51.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:51.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:51.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:51.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:51.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:51.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:51.383+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:51.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:51.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:51.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:51.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:51.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:51.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:51.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:51.629+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:52.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:52.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:52.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:52.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:52.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:52.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:52.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:52.107+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:52.649 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:52.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:52.649 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:52.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:52.649 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:52.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:52.649 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:52.362+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:52.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:52.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:52.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:52.647+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:53.536 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:53.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:53.536 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:53.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:53.536 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:53.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:53.536 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:53.362+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:53.536 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:53.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:53.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:53.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:53.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:53.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:53.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:53.117+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:53.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:53.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:53.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:53.623+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8021ab0 tx=0x7fb7f0045820 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8021ab0 tx=0x7fb7f0045820 comp rx=0 tx=0).stop 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804c190 con 0x7fb7e804d5f0 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050650 con 0x7fb8000ffcc0 2024-03-12T03:30:53.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:53.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:53.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.950+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:53.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.951+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050790 con 0x7fb7e804d5f0 2024-03-12T03:30:53.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.951+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009ba0 tx=0x7fb7f8004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:53.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de90 con 0x7fb7e804d5f0 2024-03-12T03:30:53.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f002c540 con 0x7fb7e804d5f0 2024-03-12T03:30:53.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:53.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804d5f0 2024-03-12T03:30:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:53.950+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1346 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:54.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:54.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:54.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:54.140+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:54.665 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:54.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:54.665 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:54.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:54.665 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:54.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:54.665 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:54.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:54.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:54.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:54.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:54.663+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:55.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:55.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:55.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:55.132+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:55.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:55.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:55.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:55.279+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:55.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:55.672+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:30:55.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:55.672+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:30:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:55.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:55.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:55.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:55.672+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:56.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:56.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:56.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:56.101+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:56.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:56.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:56.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:56.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:56.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:56.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:56.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:56.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:56.715+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:57.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:57.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:57.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:57.289+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:57.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:57.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:57.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:57.063+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:57.705+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:57.705+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:57.705+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:57.706+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:58.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:58.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:58.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:58.271+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:58.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:58.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:58.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:58.079+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:30:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:30:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009ba0 tx=0x7fb7f8004a70 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:30:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009ba0 tx=0x7fb7f8004a70 comp rx=0 tx=0).stop 2024-03-12T03:30:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 msgr2=0x7fb7e804ff20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:30:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d5f0 0x7fb7e804ff20 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:30:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e804d210 2024-03-12T03:30:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.950+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050650 con 0x7fb8000ffcc0 2024-03-12T03:30:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.951+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:30:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.951+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:30:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.951+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:30:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.951+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050790 con 0x7fb7e804d210 2024-03-12T03:30:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.952+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803ee50 tx=0x7fb7f000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:30:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804d210 2024-03-12T03:30:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801cda0 con 0x7fb7e804d210 2024-03-12T03:30:58.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:30:58.954+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800a300 con 0x7fb7e804d210 2024-03-12T03:30:58.974 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:30:58.950+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1409 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:30:58.974 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:58.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:58.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:58.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:58.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:58.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:30:58.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:58.713+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:30:59.267 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:59.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:59.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:59.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:59.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:59.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:30:59.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:30:59.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:30:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:59.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:59.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:59.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:30:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:30:59.266+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:59.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:59.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:59.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:30:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:30:59.759+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:00.303+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:00.303+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:00.303+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:00.304+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:00.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:00.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:00.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:00.089+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:00.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:00.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:00.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:00.804+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:01.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:01.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:01.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:01.299+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:01.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:01.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:01.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:01.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:01.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:01.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:01.073+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:01.778+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:01.778+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:01.778+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:01.779+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:02.107+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:02.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:02.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:02.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:02.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:02.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:02.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:02.305+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:02.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:02.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:02.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:02.761+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:03.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:03.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:03.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:03.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:03.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:03.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:03.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:03.308+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:03.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:03.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:03.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:03.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:03.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:03.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:03.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:03.105+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:03.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:03.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:03.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:03.735+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803ee50 tx=0x7fb7f000c520 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803ee50 tx=0x7fb7f000c520 comp rx=0 tx=0).stop 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f9e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050300 con 0x7fb7e804ff20 2024-03-12T03:31:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804f600 con 0x7fb8000ffcc0 2024-03-12T03:31:03.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.951+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:03.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.952+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:03.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.952+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:03.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.952+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804f740 con 0x7fb7e804ff20 2024-03-12T03:31:03.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.952+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009ba0 tx=0x7fb7f80490d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:03.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.954+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de90 con 0x7fb7e804ff20 2024-03-12T03:31:03.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.954+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804ff20 2024-03-12T03:31:03.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:03.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804ff20 2024-03-12T03:31:04.278 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:03.951+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1470 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:04.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:04.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:04.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:04.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:04.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:04.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:04.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:04.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:04.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:04.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:04.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:04.276+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:04.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:04.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:05.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:04.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:05.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:04.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:05.133+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:05.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:05.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:05.135+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:05.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:05.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:05.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:05.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:05.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:05.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:05.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:05.744+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:06.148+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:06.148+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:06.148+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:06.148+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:06.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:06.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:06.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:06.245+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:06.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:06.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:06.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:06.778+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:07.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:07.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:07.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:07.175+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:07.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:07.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:07.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:07.218+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:07.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:07.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:07.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:08.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:07.803+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:08.218+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:08.218+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:08.218+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:08.218+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:08.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:08.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:08.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:08.195+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:08.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:31:08.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:08.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009ba0 tx=0x7fb7f80490d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:08.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009ba0 tx=0x7fb7f80490d0 comp rx=0 tx=0).stop 2024-03-12T03:31:08.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:08.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:08.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:08.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:08.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050300 con 0x7fb7e804d210 2024-03-12T03:31:08.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.951+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb8000ffcc0 2024-03-12T03:31:08.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.952+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:08.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:08.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:08.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804fc80 con 0x7fb7e804d210 2024-03-12T03:31:08.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8052010 tx=0x7fb7f002ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:08.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.954+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804d210 2024-03-12T03:31:08.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.954+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803f6c0 con 0x7fb7e804d210 2024-03-12T03:31:08.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:08.956+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800a300 con 0x7fb7e804d210 2024-03-12T03:31:09.164 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:08.951+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1534 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:09.164 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:08.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:09.164 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:08.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:09.164 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:08.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:09.164 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:08.847+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:09.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:09.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:09.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:09.216+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:09.161+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:09.161+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:09.161+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:09.162+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:10.198 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:09.844+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:10.198 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:09.844+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:10.199 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:09.844+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:10.199 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:09.845+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:10.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:10.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:10.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:10.258+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:10.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:10.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:10.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:10.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:10.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:10.196+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:11.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:11.031+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:31:11.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:11.031+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:31:11.201 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:10.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:11.201 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:10.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:11.202 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:10.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:11.202 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:10.838+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:11.264+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:11.264+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:11.264+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:11.265+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:11.197+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:11.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:11.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:11.199+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:12.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:11.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:12.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:11.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:12.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:11.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:12.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:11.846+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:12.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:12.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:12.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:12.222+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:12.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:12.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:12.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:12.296+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:13.197 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:12.849+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:13.197 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:12.849+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:13.197 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:12.849+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:13.197 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:12.850+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:13.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.277+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e00057f0 con 0x7fb7e804d210 2024-03-12T03:31:13.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:13.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:13.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:13.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:13.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:13.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:13.537 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:13.195+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:13.537 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:13.338+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:13.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:13.338+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:13.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:13.338+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:13.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:13.338+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:13.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:31:13.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:13.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8052010 tx=0x7fb7f002ccb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:13.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8052010 tx=0x7fb7f002ccb0 comp rx=0 tx=0).stop 2024-03-12T03:31:13.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb7e804ff20 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fc80 con 0x7fb8000ffcc0 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.952+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:13.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050b90 con 0x7fb7e804ff20 2024-03-12T03:31:13.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.954+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004c040 tx=0x7fb7f000b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:13.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804ff20 2024-03-12T03:31:13.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800dd00 con 0x7fb7e804ff20 2024-03-12T03:31:13.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:13.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800a300 con 0x7fb7e804ff20 2024-03-12T03:31:14.246 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:13.952+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1598 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:14.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:13.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:14.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:13.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:14.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:13.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:14.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:13.901+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:14.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:14.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:14.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:14.245+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:14.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:14.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:14.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:14.372+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:15.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:14.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:15.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:14.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:15.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:14.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:15.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:14.854+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:15.406+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:15.406+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:15.406+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:15.407+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:15.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:15.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:15.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:15.211+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:16.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:15.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:16.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:15.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:16.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:15.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:16.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:15.837+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:16.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:16.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:16.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:16.364+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:16.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:16.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:16.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:16.252+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:17.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:16.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:17.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:16.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:17.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:16.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:17.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:16.868+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:17.400+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:17.400+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:17.400+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:17.401+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:17.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:17.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:17.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:17.252+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:18.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:17.895+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:18.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:17.895+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:18.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:17.895+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:18.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:17.896+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:18.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:18.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:18.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:18.237+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:18.376+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:18.376+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:18.376+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:18.377+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004c040 tx=0x7fb7f000b1b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004c040 tx=0x7fb7f000b1b0 comp rx=0 tx=0).stop 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:18.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80503d0 con 0x7fb7e804d210 2024-03-12T03:31:18.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb8000ffcc0 2024-03-12T03:31:18.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:18.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.954+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:18.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.954+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:18.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.954+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804fc80 con 0x7fb7e804d210 2024-03-12T03:31:18.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.954+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804e410 tx=0x7fb7f8000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:18.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803b400 con 0x7fb7e804d210 2024-03-12T03:31:18.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801f980 con 0x7fb7e804d210 2024-03-12T03:31:18.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:18.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804d210 2024-03-12T03:31:19.272 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:18.953+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1662 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:19.273 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:18.891+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:19.273 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:18.891+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:19.273 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:18.891+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:19.273 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:18.891+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:19.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:19.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:19.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:19.341+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:19.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:19.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:19.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:19.271+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:20.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:19.847+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:20.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:19.847+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:20.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:19.847+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:20.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:19.848+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:20.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:20.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:20.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:20.264+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:20.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:20.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:20.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:20.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:20.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:20.363+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:21.244 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:20.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:21.245 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:20.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:21.245 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:20.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:21.245 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:20.878+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:21.390+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:21.390+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:21.390+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:21.390+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:21.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:21.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:21.242+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:21.243+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:22.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:21.884+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:22.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:21.884+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:22.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:21.884+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:22.214 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:21.885+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:22.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:22.278+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e00057f0 con 0x7fb7e804d210 2024-03-12T03:31:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:22.348+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:22.348+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:22.348+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:22.349+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:22.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:22.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:22.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:22.212+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:23.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:22.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:23.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:22.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:23.187 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:22.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:23.187 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:22.890+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:23.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.278+0000 7fb7e77fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e00057f0 con 0x7fb7e804d210 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:23.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:23.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:23.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:23.184+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:23.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:23.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:23.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:23.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:23.311+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:23.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804e410 tx=0x7fb7f8000f40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb7f804e410 tx=0x7fb7f8000f40 comp rx=0 tx=0).stop 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80503d0 con 0x7fb7e804ff20 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.953+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb8000ffcc0 2024-03-12T03:31:23.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.954+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:23.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:23.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:23.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804fc80 con 0x7fb7e804ff20 2024-03-12T03:31:23.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.956+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800d4a0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:23.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.956+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8053460 con 0x7fb7e804ff20 2024-03-12T03:31:23.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8053460 con 0x7fb7e804ff20 2024-03-12T03:31:23.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:23.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804ff20 2024-03-12T03:31:24.181 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:23.954+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1724 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:24.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:23.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:24.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:23.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:24.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:23.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:24.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:23.872+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:24.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:24.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:24.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:24.179+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:24.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:24.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:24.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:24.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:24.332+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:25.200 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:24.827+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:25.200 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:24.827+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:25.200 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:24.827+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:25.200 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:24.827+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:25.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:25.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:25.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:25.345+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:25.197+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:25.197+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:25.197+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:25.198+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:25.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:25.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:25.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:25.793+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:26.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:26.391+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:31:26.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:26.391+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:31:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:26.384+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:26.384+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:26.384+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:26.385+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:26.235+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:26.235+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:26.235+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:26.235+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:26.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:26.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:26.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:26.791+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:27.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:27.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:27.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:27.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:27.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:27.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:27.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:27.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:27.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:27.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:27.373+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:27.749+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:27.749+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:27.749+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:27.750+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:28.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:28.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:28.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:28.415+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:28.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:28.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:28.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:28.211+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.954+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.954+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.954+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800d4a0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.954+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800d4a0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80503d0 con 0x7fb7e804d210 2024-03-12T03:31:28.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb8000ffcc0 2024-03-12T03:31:28.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:28.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:28.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:28.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804fc80 con 0x7fb7e804d210 2024-03-12T03:31:28.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.956+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8029f30 tx=0x7fb7f800d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:28.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.956+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804e020 con 0x7fb7e804d210 2024-03-12T03:31:28.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8030780 con 0x7fb7e804d210 2024-03-12T03:31:28.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:28.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804d210 2024-03-12T03:31:29.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:28.954+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1787 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:28.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:28.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:28.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:28.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:29.382+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:29.382+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:29.382+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:29.383+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:29.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:29.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:29.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:29.221+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:29.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:29.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:29.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:29.736+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:30.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:30.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:30.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:30.243+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:30.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:30.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:30.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:30.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:30.363+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:30.703+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:30.703+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:30.703+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:30.704+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:31.330+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:31.330+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:31.330+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:31.331+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:31.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:31.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:31.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:31.208+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:31.693+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:31.693+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:31.693+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:31.694+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:32.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:32.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:32.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:32.284+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:32.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:32.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:32.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:32.244+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:32.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:32.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:32.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:32.649+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:33.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:33.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:33.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:33.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:33.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:33.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:33.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:33.278+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:33.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:33.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:33.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:33.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:33.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:33.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:33.540 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:33.281+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:33.632+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:33.632+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:33.632+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:33.633+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:33.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:31:33.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:33.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8029f30 tx=0x7fb7f800d990 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:33.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8029f30 tx=0x7fb7f800d990 comp rx=0 tx=0).stop 2024-03-12T03:31:33.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:33.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:33.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:33.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:33.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb7e804ff20 2024-03-12T03:31:33.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fc80 con 0x7fb8000ffcc0 2024-03-12T03:31:33.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:33.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.956+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:33.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.956+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:33.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.956+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050b90 con 0x7fb7e804ff20 2024-03-12T03:31:33.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.956+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80308f0 tx=0x7fb7f801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:33.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804e020 con 0x7fb7e804ff20 2024-03-12T03:31:33.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8031430 con 0x7fb7e804ff20 2024-03-12T03:31:33.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:33.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804ff20 2024-03-12T03:31:34.235 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:33.955+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1851 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:34.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:34.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:34.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:34.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:34.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:34.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:34.233+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:34.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:34.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:34.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:34.245+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:34.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:34.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:34.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:34.607+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:35.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:35.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:35.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:35.217+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:35.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:35.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:35.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:35.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:35.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:35.215+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:35.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:35.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:35.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:35.609+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:36.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:36.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:36.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:36.196+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:36.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:36.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:36.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:36.177+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:36.626+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:36.626+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:36.626+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:36.627+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:37.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:37.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:37.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:37.244+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:37.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:37.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:37.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:37.161+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:37.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:37.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:37.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:37.629+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:38.272+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:38.272+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:38.272+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:38.273+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:38.202+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:38.202+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:38.202+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:38.202+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.955+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80308f0 tx=0x7fb7f801ccc0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.955+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80308f0 tx=0x7fb7f801ccc0 comp rx=0 tx=0).stop 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.956+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e8050650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.956+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e8050650 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.956+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.956+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.956+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb7e804d210 2024-03-12T03:31:38.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.956+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fc80 con 0x7fb8000ffcc0 2024-03-12T03:31:38.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.956+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:38.960 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:38.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:38.960 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:38.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:38.960 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:38.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:38.960 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:38.637+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:38.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.957+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:38.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.957+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:38.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.957+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050b90 con 0x7fb7e804d210 2024-03-12T03:31:38.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.957+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0014dd0 tx=0x7fb7f804b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:38.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de90 con 0x7fb7e804d210 2024-03-12T03:31:38.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb20 con 0x7fb7e804d210 2024-03-12T03:31:38.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:38.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804d210 2024-03-12T03:31:39.212 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:38.956+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1914 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:39.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:39.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:39.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:39.211+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:39.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:39.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:39.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:39.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:39.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:39.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:39.273+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:39.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:39.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:39.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:39.663+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:40.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:40.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:40.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:40.307+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:40.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:40.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:40.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:40.246+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:40.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:40.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:40.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:40.693+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:41.266+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:41.266+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:41.266+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:41.267+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:41.276+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:41.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:41.276+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:41.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:41.276+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:41.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:41.277+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:41.751+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:31:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:41.751+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:31:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:41.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:41.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:41.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:41.671+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:42.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:42.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:42.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:42.301+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:42.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:42.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:42.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:42.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:42.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:42.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:42.241+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:42.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:42.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:42.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:42.678+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:43.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:43.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:43.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:43.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:43.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:43.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:43.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:43.235+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:43.540 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:43.292+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:43.540 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:43.292+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:43.540 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:43.292+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:43.540 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:43.293+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:43.676+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:43.676+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:43.676+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:43.677+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:43.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.956+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:31:43.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.956+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:43.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.956+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0014dd0 tx=0x7fb7f804b8a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:43.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.956+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0014dd0 tx=0x7fb7f804b8a0 comp rx=0 tx=0).stop 2024-03-12T03:31:43.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:43.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:43.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:43.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:43.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb7e804ff20 2024-03-12T03:31:43.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fc80 con 0x7fb8000ffcc0 2024-03-12T03:31:43.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:43.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:43.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:43.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.957+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804ff20 2024-03-12T03:31:43.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.958+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000c2f0 tx=0x7fb7f0024620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:43.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de90 con 0x7fb7e804ff20 2024-03-12T03:31:43.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804ff20 2024-03-12T03:31:43.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:43.959+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804ff20 2024-03-12T03:31:44.284 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:43.957+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 1977 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:44.328+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:44.328+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:44.328+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:44.329+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:44.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:44.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:44.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:44.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:44.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:44.282+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:44.678+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:44.678+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:44.678+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:44.679+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:45.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:45.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:45.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:45.258+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:45.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:45.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:45.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:45.344+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:45.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:45.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:45.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:45.725+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:46.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:46.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:46.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:46.254+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:46.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:46.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:46.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:46.364+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:46.702+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:46.702+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:46.702+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:46.702+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:47.282+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:47.282+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:47.282+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:47.283+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:47.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:47.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:47.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:47.344+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:47.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:47.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:47.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:47.741+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:48.342+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:48.342+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:48.342+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:48.342+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:48.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:48.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:48.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:48.272+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:48.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:31:48.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:48.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000c2f0 tx=0x7fb7f0024620 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:48.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000c2f0 tx=0x7fb7f0024620 comp rx=0 tx=0).stop 2024-03-12T03:31:48.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:48.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:48.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:48.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:48.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e804d210 2024-03-12T03:31:48.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.957+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb8000ffcc0 2024-03-12T03:31:48.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:48.961 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:48.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:48.961 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:48.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:48.961 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:48.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:48.961 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:48.696+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:48.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.958+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:48.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:48.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.958+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804fc80 con 0x7fb7e804d210 2024-03-12T03:31:48.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800dcd0 tx=0x7fb7f00447e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:48.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804e020 con 0x7fb7e804d210 2024-03-12T03:31:48.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8030ec0 con 0x7fb7e804d210 2024-03-12T03:31:48.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:48.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80061e0 con 0x7fb7e804d210 2024-03-12T03:31:49.298 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:48.957+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2042 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:49.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:49.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:49.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:49.296+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:49.386+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:49.386+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:49.386+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:49.387+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:49.721+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:49.721+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:49.721+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:49.722+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:50.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:50.417+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:50.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:50.417+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:50.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:50.417+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:50.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:50.418+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:50.732 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:50.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:50.732 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:50.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:50.732 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:50.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:50.732 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:50.344+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:50.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:50.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:50.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:50.730+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:51.706 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:51.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:51.706 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:51.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:51.706 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:51.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:51.707 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:51.323+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:51.707 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:51.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:51.707 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:51.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:51.707 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:51.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:51.707 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:51.425+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:51.703+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:51.703+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:51.703+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:51.704+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:52.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:52.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:52.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:52.364+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:52.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:52.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:52.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:52.447+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:52.716+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:52.716+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:52.716+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:52.717+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:53.753 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:53.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:53.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:53.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:53.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:53.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:53.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:53.378+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:53.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:53.428+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:53.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:53.428+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:53.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:53.428+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:53.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:53.429+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:53.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.957+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.957+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800dcd0 tx=0x7fb7f00447e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800dcd0 tx=0x7fb7f00447e0 comp rx=0 tx=0).stop 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e804ff20 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb8000ffcc0 2024-03-12T03:31:53.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:53.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.959+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:53.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.959+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:53.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.959+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e804fc80 con 0x7fb7e804ff20 2024-03-12T03:31:53.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.959+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800dd20 tx=0x7fb7f804e7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:53.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80077b0 con 0x7fb7e804ff20 2024-03-12T03:31:53.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f80143d0 con 0x7fb7e804ff20 2024-03-12T03:31:53.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:53.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80284f0 con 0x7fb7e804ff20 2024-03-12T03:31:54.042 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:53.958+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2106 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:54.042 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:53.750+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:54.042 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:53.750+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:54.042 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:53.750+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:54.042 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:53.752+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:54.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:54.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:54.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:54.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:54.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:54.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:54.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:54.339+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:54.740 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:54.416+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:54.740 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:54.416+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:54.740 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:54.416+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:54.740 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:54.416+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:54.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:54.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:54.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:54.738+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:55.725 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:55.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:55.726 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:55.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:55.726 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:55.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:55.726 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:55.325+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:55.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:55.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:55.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:55.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:55.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:55.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:55.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:55.445+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:55.723+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:55.723+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:55.723+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:55.724+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:56.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:56.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:56.764 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:56.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:56.764 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:56.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:56.764 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:56.316+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:56.764 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:56.463+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:56.764 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:56.463+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:56.764 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:56.463+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:56.764 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:56.464+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:56.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:56.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:56.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:56.761+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:57.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:57.111+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:31:57.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:57.111+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:31:57.500 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:57.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:57.500 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:57.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:57.500 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:57.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:57.500 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:57.273+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:57.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:57.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:57.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:57.719+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:57.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:57.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:57.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:57.498+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:58.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:58.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:58.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:58.518+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:58.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:58.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:58.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:58.285+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:58.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:31:58.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:31:58.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800dd20 tx=0x7fb7f804e7d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:31:58.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800dd20 tx=0x7fb7f804e7d0 comp rx=0 tx=0).stop 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb7e804d210 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.958+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fc80 con 0x7fb8000ffcc0 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.959+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:31:58.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:31:58.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.959+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804d210 2024-03-12T03:31:58.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005aff0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:31:58.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de90 con 0x7fb7e804d210 2024-03-12T03:31:58.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004bce0 con 0x7fb7e804d210 2024-03-12T03:31:58.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:31:58.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804d210 2024-03-12T03:31:58.974 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:31:58.958+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2171 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:31:58.974 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:58.720+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:58.974 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:58.720+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:58.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:58.720+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:31:58.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:58.720+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:31:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:59.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:59.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:59.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:31:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:31:59.300+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:31:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:59.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:59.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:59.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:31:59.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:31:59.479+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:59.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:59.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:59.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:31:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:31:59.728+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:00.475 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:00.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:00.475 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:00.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:00.475 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:00.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:00.475 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:00.334+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:00.472+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:00.472+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:00.472+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:00.473+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:00.701+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:00.701+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:00.701+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:00.702+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:01.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:01.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:01.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:01.488+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:01.375+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:01.375+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:01.375+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:01.709 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:01.376+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:01.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:01.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:01.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:01.707+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:02.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:02.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:02.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:02.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:02.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:02.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:02.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:02.490+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:02.749 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:02.406+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:02.749 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:02.406+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:02.749 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:02.406+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:02.749 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:02.407+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:02.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:02.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:02.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:02.747+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:03.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:03.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:03.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:03.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:03.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:03.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:03.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:03.415+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:03.778 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:03.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:03.778 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:03.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:03.778 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:03.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:03.779 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:03.471+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:03.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.958+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:32:03.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.958+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:03.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005aff0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:03.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005aff0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:32:03.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:03.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:03.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:03.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:03.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fbd0 con 0x7fb7e804ff20 2024-03-12T03:32:03.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fd10 con 0x7fb8000ffcc0 2024-03-12T03:32:03.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:03.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.960+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:03.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.960+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:03.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.960+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804ff20 2024-03-12T03:32:03.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.960+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d7f0 tx=0x7fb7f0044750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:03.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de90 con 0x7fb7e804ff20 2024-03-12T03:32:03.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804ff20 2024-03-12T03:32:03.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:03.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804ff20 2024-03-12T03:32:04.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:03.959+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2235 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:04.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:03.775+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:04.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:03.775+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:04.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:03.775+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:04.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:03.776+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:04.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:04.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:04.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:04.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:04.745 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:04.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:04.745 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:04.502+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:04.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:04.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:04.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:04.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:04.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:04.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:04.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:04.397+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:04.741+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:04.741+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:04.741+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:04.742+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:05.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:05.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:05.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:05.550+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:05.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:05.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:05.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:05.794 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:05.426+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:05.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:05.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:05.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:05.792+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:06.510+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:06.510+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:06.510+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:06.511+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:06.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:06.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:06.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:06.767 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:06.455+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:06.764+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:06.764+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:06.764+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:06.764+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:07.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:07.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:07.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:07.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:07.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:07.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:07.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:07.506+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:07.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:07.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:07.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:07.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:07.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:07.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:07.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:07.489+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:07.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:07.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:07.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:07.801+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:08.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:08.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:08.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:08.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:08.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:08.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:08.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:08.538+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:08.789 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:08.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:08.789 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:08.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:08.789 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:08.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:08.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:08.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:08.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:08.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:08.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:08.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:08.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:08.495+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d7f0 tx=0x7fb7f0044750 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d7f0 tx=0x7fb7f0044750 comp rx=0 tx=0).stop 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff20 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fbd0 con 0x7fb7e804d210 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fd10 con 0x7fb8000ffcc0 2024-03-12T03:32:08.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.959+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:08.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.960+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:08.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:08.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.960+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804d210 2024-03-12T03:32:08.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e30 tx=0x7fb7f002de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:08.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80077b0 con 0x7fb7e804d210 2024-03-12T03:32:08.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8007aa0 con 0x7fb7e804d210 2024-03-12T03:32:08.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:08.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8006080 con 0x7fb7e804d210 2024-03-12T03:32:09.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:08.959+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2301 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:08.787+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:09.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:09.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:09.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:09.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:09.777 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:09.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:09.777 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:09.452+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:09.777 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:09.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:09.777 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:09.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:09.777 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:09.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:09.777 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:09.545+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:09.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:09.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:09.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:09.774+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:10.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:10.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:10.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:10.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:10.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:10.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:10.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:10.565+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:10.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:10.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:10.777 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:10.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:10.777 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:10.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:10.777 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:10.452+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:10.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:10.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:10.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:10.774+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:11.807 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:11.517+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:11.808 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:11.517+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:11.808 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:11.517+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:11.808 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:11.518+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:11.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:11.428+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:11.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:11.428+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:11.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:11.428+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:11.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:11.428+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:11.805+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:11.805+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:11.805+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:11.806+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:12.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:12.471+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:32:12.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:12.471+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:32:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:12.487+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:12.487+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:12.487+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:12.488+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:12.435+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:12.435+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:12.435+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:12.436+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:12.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:12.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:12.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:12.836+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:13.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:13.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:13.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:13.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:13.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:13.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:13.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:13.517+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:13.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:13.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:13.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:13.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:13.463+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:13.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.959+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e30 tx=0x7fb7f002de40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e30 tx=0x7fb7f002de40 comp rx=0 tx=0).stop 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 msgr2=0x7fb7e804f600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804d210 0x7fb7e804f600 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804ff60 con 0x7fb7e804fb40 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb8000ffcc0 2024-03-12T03:32:13.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:13.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:13.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:13.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.960+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80501e0 con 0x7fb7e804fb40 2024-03-12T03:32:13.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.961+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e80504a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f8014df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:13.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80077b0 con 0x7fb7e804fb40 2024-03-12T03:32:13.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8017ce0 con 0x7fb7e804fb40 2024-03-12T03:32:13.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:13.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804fb40 2024-03-12T03:32:14.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:13.960+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2365 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:13.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:13.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:13.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:13.808+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:14.503+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:14.503+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:14.503+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:14.504+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:14.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:14.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:14.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:14.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:14.502+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:14.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:14.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:14.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:14.787+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:15.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:15.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:15.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:15.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:15.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:15.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:15.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:15.519+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:15.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:15.466+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:15.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:15.466+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:15.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:15.466+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:15.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:15.466+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:15.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:15.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:15.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:15.835+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:16.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:16.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:16.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:16.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:16.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:16.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:16.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:16.495+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:16.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:16.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:16.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:16.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:16.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:16.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:16.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:16.451+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:16.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:16.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:16.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:16.796+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:17.782 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:17.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:17.782 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:17.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:17.782 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:17.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:17.782 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:17.493+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:17.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:17.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:17.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:17.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:17.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:17.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:17.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:17.468+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:17.779+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:17.779+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:17.779+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:17.780+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:18.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:18.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:18.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:18.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:18.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:18.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:18.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:18.498+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:18.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:18.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:18.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:18.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:18.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:18.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:18.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:18.502+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:18.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:32:18.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:18.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e80504a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f8014df0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:18.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e80504a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f8014df0 comp rx=0 tx=0).stop 2024-03-12T03:32:18.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:18.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:18.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:18.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:18.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804f760 con 0x7fb7e804ff60 2024-03-12T03:32:18.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804f8a0 con 0x7fb8000ffcc0 2024-03-12T03:32:18.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.960+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:18.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.961+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:18.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.961+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:18.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.961+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804ff60 2024-03-12T03:32:18.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 0x7fb7e804f220 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800b0e0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:18.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80077b0 con 0x7fb7e804ff60 2024-03-12T03:32:18.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8017ce0 con 0x7fb7e804ff60 2024-03-12T03:32:18.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:18.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804ff60 2024-03-12T03:32:19.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:18.960+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2428 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:19.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:18.750+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:19.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:18.750+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:19.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:18.750+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:19.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:18.751+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:19.796 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:19.467+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:19.796 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:19.467+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:19.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:19.467+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:19.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:19.468+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:19.797 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:19.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:19.797 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:19.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:19.797 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:19.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:19.797 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:19.533+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:19.793+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:19.793+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:19.793+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:19.795+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:20.766 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:20.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:20.766 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:20.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:20.766 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:20.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:20.766 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:20.538+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:20.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:20.464+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:20.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:20.464+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:20.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:20.464+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:20.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:20.464+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:20.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:20.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:20.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:20.764+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:21.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:21.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:21.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:21.728+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:21.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:21.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:21.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:21.486+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:21.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:21.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:21.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:21.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:21.814 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:21.586+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:22.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:22.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:22.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:22.539+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:22.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:22.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:22.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:22.681+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:22.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:22.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:22.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:22.514+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:23.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:23.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:23.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:23.495+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:23.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:23.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:23.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:23.566+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:23.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:23.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:23.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:23.686+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 0x7fb7e804f220 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800b0e0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 0x7fb7e804f220 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800b0e0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804ff60 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:23.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:23.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fb40 con 0x7fb7e804f760 2024-03-12T03:32:23.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804fc80 con 0x7fb8000ffcc0 2024-03-12T03:32:23.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.961+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:23.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.962+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:23.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.962+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:23.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.962+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:32:23.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.962+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8018010 tx=0x7fb7f8037000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:23.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804f760 2024-03-12T03:32:23.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb00 con 0x7fb7e804f760 2024-03-12T03:32:23.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:23.964+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071070 con 0x7fb7e804f760 2024-03-12T03:32:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:23.961+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2490 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:24.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:24.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:24.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:24.580+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:24.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:24.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:24.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:24.526+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:24.694+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:24.694+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:24.694+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:24.695+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:25.543+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:25.543+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:25.543+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:25.544+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:25.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:25.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:25.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:25.484+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:25.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:25.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:25.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:25.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:25.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:25.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:25.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:26.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:26.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:26.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:26.458+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:26.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:26.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:26.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:26.687+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:26.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:26.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:26.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:26.593+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:27.719 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:27.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:27.719 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:27.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:27.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:27.586+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:27.654+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:27.654+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:27.654+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:27.655+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:27.442+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:27.442+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:27.720 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:27.442+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:27.721 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:27.443+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:27.831+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:32:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:27.831+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:32:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:28.597+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:28.597+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:28.597+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:28.598+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:28.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:28.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:28.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:28.641+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:28.394+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:28.394+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:28.394+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:28.395+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:28.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.961+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:32:28.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.961+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.961+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8018010 tx=0x7fb7f8037000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.961+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8018010 tx=0x7fb7f8037000 comp rx=0 tx=0).stop 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.961+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.961+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.961+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.962+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050030 con 0x7fb7e804fb40 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050170 con 0x7fb8000ffcc0 2024-03-12T03:32:28.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.962+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.962+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:32:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.963+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8019300 tx=0x7fb7f000b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804fb40 2024-03-12T03:32:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804fb40 2024-03-12T03:32:28.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:28.964+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804fb40 2024-03-12T03:32:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:28.962+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2554 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:29.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:29.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:29.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:29.430+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:29.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:29.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:29.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:29.587+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:29.625+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:29.625+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:29.625+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:29.626+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:30.619+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:30.619+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:30.619+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:30.619+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:30.431+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:30.431+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:30.431+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:30.432+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:30.638+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:30.638+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:30.638+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:30.639+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:31.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:31.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:31.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:31.629+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:31.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:31.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:31.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:31.438+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:31.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:31.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:31.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:31.662+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:32.775 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:32.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:32.775 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:32.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:32.775 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:32.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:32.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:32.598+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:32.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:32.439+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:32.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:32.439+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:32.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:32.439+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:32.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:32.440+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:32.776 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:32.635+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:32.776 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:32.635+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:32.777 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:32.635+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:32.777 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:32.636+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:33.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:33.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:33.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:33.604+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:33.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:33.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:33.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:33.581+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:33.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:33.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:33.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:33.460+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8019300 tx=0x7fb7f000b1b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8019300 tx=0x7fb7f000b1b0 comp rx=0 tx=0).stop 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:33.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:33.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050030 con 0x7fb7e804f760 2024-03-12T03:32:33.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050170 con 0x7fb8000ffcc0 2024-03-12T03:32:33.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:33.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:33.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:33.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:32:33.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.962+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dcc0 tx=0x7fb7f804ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:33.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804f760 2024-03-12T03:32:33.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804f760 2024-03-12T03:32:33.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:33.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804f760 2024-03-12T03:32:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:33.962+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2619 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:34.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:34.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:34.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:34.608+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:34.449+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:34.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:34.449+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:34.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:34.449+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:34.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:34.450+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:34.578+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:34.578+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:34.578+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:34.579+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:35.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:35.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:35.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:35.636+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:35.411+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:35.411+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:35.411+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:35.412+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:35.609+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:35.609+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:35.609+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:35.610+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:36.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:36.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:36.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:36.375+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:36.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:36.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:36.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:36.575+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:36.595+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:36.595+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:36.595+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:36.596+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:37.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:37.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:37.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:37.342+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:37.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:37.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:37.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:37.527+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:37.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:37.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:37.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:37.586+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:38.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:38.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:38.316+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:38.317+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:38.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:38.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:38.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:38.494+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:38.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:38.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:38.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:38.556+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:38.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.962+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dcc0 tx=0x7fb7f804ac60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dcc0 tx=0x7fb7f804ac60 comp rx=0 tx=0).stop 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050030 con 0x7fb7e804fb40 2024-03-12T03:32:38.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.963+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050170 con 0x7fb8000ffcc0 2024-03-12T03:32:38.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.964+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:38.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.964+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:38.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.964+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:38.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.964+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:32:38.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.965+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8008be0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:38.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804fb40 2024-03-12T03:32:38.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb00 con 0x7fb7e804fb40 2024-03-12T03:32:38.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:38.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:32:39.285 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:38.963+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2686 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:39.541 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:39.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:39.541 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:39.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:39.541 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:39.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:39.541 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:39.283+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:39.542 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:39.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:39.542 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:39.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:39.542 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:39.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:39.542 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:39.467+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:39.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:39.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:39.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:39.539+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:40.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:40.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:40.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:40.439+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:40.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:40.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:40.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:40.306+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:40.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:40.306+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:40.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:40.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:40.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:40.566+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:41.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:41.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:41.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:41.418+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:41.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:41.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:41.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:41.316+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:41.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:41.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:41.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:41.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:41.551+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:42.337+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:42.337+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:42.337+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:42.338+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:42.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:42.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:42.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:42.468+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:42.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:42.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:42.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:42.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:42.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:42.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:42.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:42.569+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:43.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.191+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:32:43.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.191+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:32:43.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:43.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:43.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:43.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:43.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:43.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:43.481 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:43.315+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:43.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:43.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:43.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:43.479+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:43.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:43.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:43.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:43.593+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:43.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:32:43.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8008be0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8008be0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050030 con 0x7fb7e804f760 2024-03-12T03:32:43.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050170 con 0x7fb8000ffcc0 2024-03-12T03:32:43.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.964+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:43.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.965+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:43.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.965+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:43.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.965+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:32:43.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.965+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8040270 tx=0x7fb7f004c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:43.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8037de0 con 0x7fb7e804f760 2024-03-12T03:32:43.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804f760 2024-03-12T03:32:43.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:43.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804f760 2024-03-12T03:32:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:43.965+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2750 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:44.592 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:44.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:44.592 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:44.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:44.592 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:44.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:44.592 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:44.326+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:44.593 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:44.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:44.593 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:44.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:44.593 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:44.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:44.593 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:44.452+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:44.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:44.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:44.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:44.590+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:45.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:45.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:45.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:45.411+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:45.287+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:45.287+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:45.287+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:45.288+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:45.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:45.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:45.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:45.582+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:46.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:46.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:46.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:46.290+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:46.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:46.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:46.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:46.459+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:46.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:46.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:46.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:46.586+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:47.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:47.317+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:47.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:47.317+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:47.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:47.317+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:47.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:47.318+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:47.599 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:47.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:47.600 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:47.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:47.600 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:47.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:47.600 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:47.440+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:47.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:47.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:47.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:47.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:47.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:47.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:47.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:47.597+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:48.632 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:48.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:48.632 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:48.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:48.633 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:48.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:48.633 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:48.336+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:48.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:48.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:48.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:48.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:48.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:48.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:48.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:48.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8040270 tx=0x7fb7f004c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8040270 tx=0x7fb7f004c000 comp rx=0 tx=0).stop 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050030 con 0x7fb7e804fb40 2024-03-12T03:32:48.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050170 con 0x7fb8000ffcc0 2024-03-12T03:32:48.969 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:48.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:48.969 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:48.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:48.969 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:48.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:48.969 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:48.631+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:48.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.965+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:48.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.966+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:48.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.966+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:48.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.966+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:32:48.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.966+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8008be0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:48.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8037de0 con 0x7fb7e804fb40 2024-03-12T03:32:48.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801d4a0 con 0x7fb7e804fb40 2024-03-12T03:32:48.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:48.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804fb40 2024-03-12T03:32:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:48.965+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2821 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:49.629 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:49.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:49.630 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:49.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:49.630 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:49.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:49.630 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:49.354+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:49.630 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:49.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:49.630 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:49.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:49.630 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:49.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:49.630 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:49.402+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:49.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:49.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:49.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:49.627+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:50.633 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:50.373+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:50.633 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:50.373+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:50.633 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:50.373+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:50.633 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:50.374+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:50.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:50.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:50.634 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:50.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:50.634 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:50.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:50.634 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:50.396+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:50.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:50.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:50.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:50.631+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:51.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:51.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:51.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:51.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:51.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:51.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:51.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:51.401+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:51.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:51.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:51.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:51.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:51.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:51.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:51.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:51.353+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:51.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:51.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:51.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:51.656+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:52.594 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:52.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:52.594 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:52.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:52.595 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:52.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:52.595 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:52.319+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:52.595 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:52.595 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:52.595 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:52.595 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:52.381+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:53.002 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:52.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:53.002 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:52.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:53.002 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:52.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:53.002 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:52.637+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:53.545 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:53.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:53.545 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:53.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:53.545 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:53.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:53.546 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:53.394+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:53.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:53.272+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:53.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:53.272+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:53.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:53.272+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:53.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:53.273+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:53.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:53.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:53.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:53.665+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:53.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8008be0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8008be0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050080 con 0x7fb7e804f760 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.965+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80501c0 con 0x7fb8000ffcc0 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.966+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.966+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.966+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:53.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.966+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:32:53.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.966+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801cde0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:53.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8037de0 con 0x7fb7e804f760 2024-03-12T03:32:53.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801d4a0 con 0x7fb7e804f760 2024-03-12T03:32:53.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:53.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804f760 2024-03-12T03:32:54.284 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:53.966+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2889 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:54.281+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:54.281+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:54.281+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:54.282+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:54.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:54.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:54.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:54.376+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:54.702+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:54.702+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:54.702+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:54.703+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:55.676 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:55.330+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:55.676 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:55.330+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:55.676 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:55.330+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:55.676 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:55.331+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:55.677 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:55.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:55.677 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:55.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:55.677 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:55.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:55.677 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:55.323+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:55.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:55.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:55.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:55.674+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:56.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:56.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:56.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:56.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:56.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:56.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:56.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:56.376+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:56.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:56.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:56.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:56.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:56.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:56.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:56.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:56.311+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:56.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:56.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:56.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:56.631+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:57.265+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:57.265+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:57.265+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:57.266+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:57.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:57.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:57.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:57.359+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:58.057 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:57.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:58.057 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:57.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:58.057 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:57.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:58.057 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:57.665+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:58.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:58.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:58.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:58.278+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:58.627 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:58.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:58.627 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:58.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:58.627 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:58.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:58.627 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:58.344+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:32:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:32:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:32:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801cde0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:32:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801cde0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:32:58.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:32:58.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:58.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:58.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:32:58.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050270 con 0x7fb7e804fb40 2024-03-12T03:32:58.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.966+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb8000ffcc0 2024-03-12T03:32:58.970 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:58.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:58.970 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:58.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:58.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:58.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:32:58.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:58.625+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:58.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.967+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:32:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.967+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:32:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.967+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:32:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.967+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050b20 con 0x7fb7e804fb40 2024-03-12T03:32:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.967+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003ffc0 tx=0x7fb7f804b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:32:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804fb40 2024-03-12T03:32:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb00 con 0x7fb7e804fb40 2024-03-12T03:32:58.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:32:58.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:32:59.303 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:32:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:32:58.966+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 2952 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:32:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:59.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:59.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:59.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:32:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:32:59.308+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:32:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:59.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:59.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:59.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:32:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:32:59.301+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:59.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:59.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:59.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:32:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:32:59.587+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:00.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:00.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:00.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:00.321+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:00.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:00.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:00.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:00.545+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:00.329+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:00.329+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:00.329+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:00.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:00.330+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:01.580 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:01.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:01.580 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:01.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:01.580 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:01.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:01.580 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:01.344+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:01.580 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:01.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:01.581 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:01.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:01.581 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:01.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:01.581 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:01.313+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:01.623+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:33:01.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:01.623+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:33:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:01.577+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:01.577+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:01.577+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:01.578+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:02.570+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:02.570+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:02.570+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:02.571+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:02.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:02.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:02.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:02.329+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:02.332+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:02.332+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:02.332+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:02.333+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:03.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:03.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:03.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:03.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:03.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:03.286+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:03.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:03.287+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:03.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:03.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:03.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:03.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:03.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:03.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:03.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:03.314+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:03.546+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:03.546+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:03.546+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:03.547+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:03.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-12T03:33:03.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:03.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003ffc0 tx=0x7fb7f804b470 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:03.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003ffc0 tx=0x7fb7f804b470 comp rx=0 tx=0).stop 2024-03-12T03:33:03.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:03.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:03.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804ffd0 con 0x7fb7e804f760 2024-03-12T03:33:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb8000ffcc0 2024-03-12T03:33:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.967+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.968+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050b20 con 0x7fb7e804f760 2024-03-12T03:33:03.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.968+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8053e20 tx=0x7fb7f0053880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:03.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8037de0 con 0x7fb7e804f760 2024-03-12T03:33:03.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801d4a0 con 0x7fb7e804f760 2024-03-12T03:33:03.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:03.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804f760 2024-03-12T03:33:04.270 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:03.967+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3014 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:04.534 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:04.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:04.534 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:04.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:04.534 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:04.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:04.534 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:04.268+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:04.535 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:04.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:04.535 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:04.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:04.535 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:04.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:04.535 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:04.321+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:04.531+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:04.531+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:04.531+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:04.532+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:05.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:05.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:05.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:05.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:05.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:05.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:05.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:05.279+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:05.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:05.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:05.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:05.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:05.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:05.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:05.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:05.241+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:05.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:05.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:05.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:05.557+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:06.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:06.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:06.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:06.198+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:06.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:06.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:06.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:06.321+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:06.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:06.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:06.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:06.604+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:07.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:07.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:07.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:07.323+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:07.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:07.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:07.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:07.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:07.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:07.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:07.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:07.171+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:07.583+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:07.583+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:07.583+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:07.584+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:08.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:08.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:08.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:08.288+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:08.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:08.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:08.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:08.144+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.967+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.967+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.967+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8053e20 tx=0x7fb7f0053880 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.967+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8053e20 tx=0x7fb7f0053880 comp rx=0 tx=0).stop 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.967+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.967+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.968+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.968+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804ffd0 con 0x7fb7e804fb40 2024-03-12T03:33:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050110 con 0x7fb8000ffcc0 2024-03-12T03:33:08.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:08.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:08.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:08.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:08.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:08.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:08.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:08.564+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.968+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.968+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.969+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.969+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.969+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003df20 tx=0x7fb7f801deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804fb40 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004cb00 con 0x7fb7e804fb40 2024-03-12T03:33:08.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:08.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071070 con 0x7fb7e804fb40 2024-03-12T03:33:09.275 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:08.968+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3077 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:09.275 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:09.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:09.275 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:09.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:09.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:09.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:09.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:09.104+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:09.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:09.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:09.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:09.273+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:10.058 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:09.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:10.058 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:09.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:10.058 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:09.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:10.058 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:09.572+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:10.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:10.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:10.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:10.056+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:10.227+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:10.227+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:10.227+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:10.227+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:10.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:10.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:10.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:10.618+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:11.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:11.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:11.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:11.234+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:11.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:11.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:11.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:11.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:11.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:11.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:11.061+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:11.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:11.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:11.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:11.615+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:12.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:12.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:12.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:12.110+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:12.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:12.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:12.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:12.201+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:12.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:12.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:12.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:12.664+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:13.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:13.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:13.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:13.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:13.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:13.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:13.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:13.061+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:13.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:13.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:13.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:13.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:13.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:13.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:13.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:13.176+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:13.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:13.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:13.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:13.706+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003df20 tx=0x7fb7f801deb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003df20 tx=0x7fb7f801deb0 comp rx=0 tx=0).stop 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb7e804f760 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.968+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80501e0 con 0x7fb8000ffcc0 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.969+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.969+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.969+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:13.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.969+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:33:13.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.969+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8015b70 tx=0x7fb7f002b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:13.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804f760 2024-03-12T03:33:13.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f002f2c0 con 0x7fb7e804f760 2024-03-12T03:33:13.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:13.972+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:33:14.180 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:13.968+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3144 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:14.180 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:14.033+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:14.180 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:14.033+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:14.181 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:14.033+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:14.181 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:14.034+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:14.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:14.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:14.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:14.178+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:15.037 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:14.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:15.037 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:14.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:15.037 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:14.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:15.037 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:14.670+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:15.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:15.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:15.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:15.035+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:15.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:15.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:15.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:15.190+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:16.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:15.681+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:16.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:15.681+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:16.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:15.681+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:16.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:15.682+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:16.057+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:16.057+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:16.057+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:16.058+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:16.164+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:16.164+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:16.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:16.164+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:16.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:16.165+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:17.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:17.047+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:33:17.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:17.047+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:33:17.050 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:16.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:17.050 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:16.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:17.050 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:16.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:17.050 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:16.676+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:17.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:17.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:17.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:17.200+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:17.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:17.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:17.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:17.065+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:17.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:17.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:17.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:17.718+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:18.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:18.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:18.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:18.105+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:18.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:18.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:18.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:18.195+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8015b70 tx=0x7fb7f002b4a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8015b70 tx=0x7fb7f002b4a0 comp rx=0 tx=0).stop 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb7e804fb40 2024-03-12T03:33:18.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80501e0 con 0x7fb8000ffcc0 2024-03-12T03:33:18.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:18.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:18.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:18.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:18.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:18.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:18.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:18.714+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:18.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.969+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:18.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.970+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:18.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.970+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:18.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.970+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:33:18.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.970+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801e000 tx=0x7fb7f00090e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:18.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8037de0 con 0x7fb7e804fb40 2024-03-12T03:33:18.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801d4a0 con 0x7fb7e804fb40 2024-03-12T03:33:18.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:18.972+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804fb40 2024-03-12T03:33:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:18.969+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3220 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:19.089+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:19.089+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:19.089+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:19.090+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:19.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:19.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:19.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:19.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:19.171+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:19.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:19.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:19.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:19.685+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:20.102+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:20.102+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:20.102+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:20.103+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:20.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:20.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:20.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:20.154+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:20.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:20.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:20.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:20.676+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:21.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:21.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:21.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:21.171+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:21.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:21.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:21.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:21.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:21.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:21.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:21.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:21.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:21.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:21.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:21.658+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:22.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:22.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:22.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:22.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:22.190+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:22.190+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:22.190+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:22.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:22.191+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:22.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:22.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:22.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:22.634+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:23.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:23.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:23.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:23.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:23.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:23.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:23.548 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:23.106+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:23.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:23.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:23.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:23.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:23.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:23.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:23.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:23.172+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:23.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:23.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:23.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:23.684+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801e000 tx=0x7fb7f00090e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801e000 tx=0x7fb7f00090e0 comp rx=0 tx=0).stop 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:23.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:23.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8048eb0 con 0x7fb7e804f760 2024-03-12T03:33:23.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.969+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb8000ffcc0 2024-03-12T03:33:23.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.970+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:23.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.970+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:23.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.970+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:23.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.970+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80501e0 con 0x7fb7e804f760 2024-03-12T03:33:23.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.971+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8034340 tx=0x7fb7f8031aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:23.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804f760 2024-03-12T03:33:23.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.972+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f002f2c0 con 0x7fb7e804f760 2024-03-12T03:33:23.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:23.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804f760 2024-03-12T03:33:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:23.970+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3292 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:24.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:24.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:24.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:24.179+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:24.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:24.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:24.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:24.079+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:24.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:24.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:24.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:24.734+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:25.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:25.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:25.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:25.098+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:25.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:25.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:25.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:25.168+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:25.756+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:25.756+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:25.756+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:25.757+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:26.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:26.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:26.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:26.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:26.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:26.192+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:26.192+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:26.192+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:26.193+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:26.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:26.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:26.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:26.747+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:27.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:27.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:27.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:27.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:27.241+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:27.241+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:27.241+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:27.242+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:27.710+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:27.710+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:27.710+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:27.711+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:28.398 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:28.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:28.399 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:28.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:28.399 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:28.129+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:28.399 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:28.130+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:28.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:28.198+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:28.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:28.198+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:28.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:28.198+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:28.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:28.200+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:28.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:33:28.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:28.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8034340 tx=0x7fb7f8031aa0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:28.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8034340 tx=0x7fb7f8031aa0 comp rx=0 tx=0).stop 2024-03-12T03:33:28.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:28.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:28.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:28.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:28.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050130 con 0x7fb7e804fb40 2024-03-12T03:33:28.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050270 con 0x7fb8000ffcc0 2024-03-12T03:33:28.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.970+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:28.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.971+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:28.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.971+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:28.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.971+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:33:28.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.971+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80426e0 tx=0x7fb7f000bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:28.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.972+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8007aa0 con 0x7fb7e804fb40 2024-03-12T03:33:28.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.972+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8007aa0 con 0x7fb7e804fb40 2024-03-12T03:33:28.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:28.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c880 con 0x7fb7e804fb40 2024-03-12T03:33:29.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:28.970+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3355 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:28.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:28.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:28.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:28.761+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:29.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:29.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:29.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:29.096+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:29.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:29.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:29.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:29.244+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:29.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:29.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:29.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:29.802+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:30.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:30.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:30.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:30.097+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:30.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:30.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:30.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:30.254+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:30.770+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:30.770+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:30.770+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:30.770+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:31.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:31.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:31.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:31.062+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:31.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:31.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:31.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:31.274+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:31.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:31.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:31.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:31.821+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:32.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:32.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:32.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:32.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:32.017+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:32.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:32.407+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:33:32.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:32.407+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:33:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:32.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:32.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:32.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:32.308+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:33.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:32.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:33.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:32.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:33.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:32.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:33.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:32.828+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:33.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:33.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:33.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:33.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:33.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:33.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:33.015+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:33.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:33.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:33.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:33.308+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:33.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80426e0 tx=0x7fb7f000bc90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80426e0 tx=0x7fb7f000bc90 comp rx=0 tx=0).stop 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e804f760 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050130 con 0x7fb8000ffcc0 2024-03-12T03:33:33.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.971+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:33.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.972+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:33.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.972+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:33.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.972+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:33:33.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.972+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801afe0 tx=0x7fb7f8009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:33.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.972+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804f760 2024-03-12T03:33:33.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.972+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f80273c0 con 0x7fb7e804f760 2024-03-12T03:33:33.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:33.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c820 con 0x7fb7e804f760 2024-03-12T03:33:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:33.971+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3417 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:33.850+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:33.850+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:33.850+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:33.850+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:34.058+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:34.058+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:34.058+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:34.059+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:34.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:34.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:34.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:34.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:34.804 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:34.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:34.804 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:34.345+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.010+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:34.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:35.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:34.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:35.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:34.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:35.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:34.801+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:35.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:35.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:35.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:35.369+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:35.976+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:35.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:35.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:35.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:35.813+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:36.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:36.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:36.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:36.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:36.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:36.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:36.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:36.405+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:36.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:36.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:36.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:36.973+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:37.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:36.776+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:37.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:36.776+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:37.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:36.776+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:37.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:36.777+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:37.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:37.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:37.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:37.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:37.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:37.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:37.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:37.424+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:38.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:37.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:38.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:37.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:38.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:37.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:38.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:37.756+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.010+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:38.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:38.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:38.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:38.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:38.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:38.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:38.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:38.375+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:33:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801afe0 tx=0x7fb7f8009830 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fb7f801afe0 tx=0x7fb7f8009830 comp rx=0 tx=0).stop 2024-03-12T03:33:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804cc70 con 0x7fb7e804fb40 2024-03-12T03:33:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.971+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050130 con 0x7fb8000ffcc0 2024-03-12T03:33:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.972+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:38.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:38.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:38.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:38.715+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:38.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.973+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:38.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.973+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:38.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.973+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:33:38.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.974+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8014ac0 tx=0x7fb7f8028430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.976+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f80273c0 con 0x7fb7e804fb40 2024-03-12T03:33:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.976+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804fb40 2024-03-12T03:33:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:38.976+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c820 con 0x7fb7e804fb40 2024-03-12T03:33:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:38.972+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3478 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:38.981+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:39.740 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:39.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:39.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:39.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:39.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:39.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:39.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:39.360+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:39.738+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:39.738+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:39.738+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:39.739+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:39.966+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:39.966+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:40.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:39.967+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:40.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:39.967+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:40.696 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:40.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:40.696 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:40.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:40.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:40.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:40.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:40.405+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:40.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:40.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:40.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:40.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:40.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:40.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:40.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:40.695+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:40.981+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:40.981+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:40.981+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:40.982+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:41.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:41.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:41.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:41.668+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:41.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:41.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:41.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:41.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:41.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:41.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:41.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:41.948+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:42.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:42.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:42.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:42.643+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:42.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:42.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:42.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:42.469+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:42.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:42.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:42.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:42.965+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:43.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:43.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:43.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:43.468+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:43.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:43.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:43.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:43.619+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:43.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.972+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.972+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.972+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8014ac0 tx=0x7fb7f8028430 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.972+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8014ac0 tx=0x7fb7f8028430 comp rx=0 tx=0).stop 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8049580 con 0x7fb7e804f760 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050130 con 0x7fb8000ffcc0 2024-03-12T03:33:43.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:43.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:43.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:43.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.973+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:33:43.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.974+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8014a60 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:43.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.974+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8052ca0 con 0x7fb7e804f760 2024-03-12T03:33:43.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.974+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800db60 con 0x7fb7e804f760 2024-03-12T03:33:43.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:43.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800d7a0 con 0x7fb7e804f760 2024-03-12T03:33:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:43.973+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3540 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:43.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:43.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:43.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:43.941+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:44.584 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:44.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:44.584 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:44.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:44.584 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:44.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:44.584 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:44.464+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:44.982 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:44.581+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:44.982 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:44.581+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:44.982 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:44.581+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:44.982 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:44.582+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:44.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:44.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:44.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:44.980+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:45.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:45.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:45.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:45.629+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:45.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:45.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:45.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:45.439+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.020+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.020+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.020+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.021+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:46.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:46.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:46.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:46.411+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:46.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:46.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:46.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:46.629+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.994+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.994+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.994+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:46.995+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:47.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:47.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:47.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:47.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:47.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:47.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:47.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:47.379+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:47.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:47.767+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:33:47.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:47.767+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:33:47.955 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:47.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:47.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:47.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:47.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:47.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:47.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:47.643+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:47.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:47.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:47.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:47.953+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:48.664 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:48.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:48.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:48.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:48.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:48.412+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:48.915 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:48.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:48.915 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:48.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:48.915 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:48.662+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:48.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:33:48.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8014a60 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8014a60 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80437a0 con 0x7fb7e804fb40 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb8000ffcc0 2024-03-12T03:33:48.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.973+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.974+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.974+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.974+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80501e0 con 0x7fb7e804fb40 2024-03-12T03:33:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.975+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0048180 tx=0x7fb7f8009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804fb40 2024-03-12T03:33:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.976+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804fb40 2024-03-12T03:33:48.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:48.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:33:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:48.973+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3602 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:48.912+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:48.912+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:48.912+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:48.913+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:49.636 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:49.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:49.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:49.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:49.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:49.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:49.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:49.371+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:49.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:49.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:49.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:49.875+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:49.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:49.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:49.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:49.635+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:50.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:50.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:50.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:50.618+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:50.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:50.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:50.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:50.403+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:50.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:50.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:50.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:50.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:51.632+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:51.632+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:51.632+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:51.633+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:51.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:51.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:51.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:51.398+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:51.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:51.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:51.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:51.927+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:52.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:52.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:52.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:52.427+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:52.588+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:52.588+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:52.588+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:52.589+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:52.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:52.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:52.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:52.885+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:53.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:53.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:53.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:53.437+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:53.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:53.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:53.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:53.613+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:53.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:33:53.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:53.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0048180 tx=0x7fb7f8009830 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:53.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0048180 tx=0x7fb7f8009830 comp rx=0 tx=0).stop 2024-03-12T03:33:53.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:53.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:53.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:53.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:53.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8043780 con 0x7fb7e804f760 2024-03-12T03:33:53.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb8000ffcc0 2024-03-12T03:33:53.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.974+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:53.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.975+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:53.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.975+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:53.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.975+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80501e0 con 0x7fb7e804f760 2024-03-12T03:33:53.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.975+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0012ba0 tx=0x7fb7f0036e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:53.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804f760 2024-03-12T03:33:53.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804f760 2024-03-12T03:33:53.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:53.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:33:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:53.974+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3665 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:53.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:53.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:53.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:53.838+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:54.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:54.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:54.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:54.566+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:54.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:54.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:54.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:54.464+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:54.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:54.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:54.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:54.817+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:55.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:55.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:55.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:55.432+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:55.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:55.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:55.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:55.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:55.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:55.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:55.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:55.617+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:55.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:55.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:55.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:55.857+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:56.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:56.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:56.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:56.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:56.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:56.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:56.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:56.385+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:56.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:56.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:56.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:56.643+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:56.843+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:56.843+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:56.843+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:56.843+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:57.658+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:57.658+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:57.658+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:57.658+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:57.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:57.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:57.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:57.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:57.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:57.420+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:57.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:57.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:57.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:57.845+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:58.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:58.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:58.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:58.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:58.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:58.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:58.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:58.467+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:33:58.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:58.632+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:58.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:58.632+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:58.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:58.632+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:58.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:58.633+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:58.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.974+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:33:58.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.974+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:33:58.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.974+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0012ba0 tx=0x7fb7f0036e10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:33:58.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.974+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0012ba0 tx=0x7fb7f0036e10 comp rx=0 tx=0).stop 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.975+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.975+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.975+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8043780 con 0x7fb7e804fb40 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb8000ffcc0 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.975+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.976+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.976+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.976+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80501e0 con 0x7fb7e804fb40 2024-03-12T03:33:58.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.977+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003d370 tx=0x7fb7f004cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:33:58.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.979+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0031ef0 con 0x7fb7e804fb40 2024-03-12T03:33:58.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.979+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a4c0 con 0x7fb7e804fb40 2024-03-12T03:33:58.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:33:58.980+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804fb40 2024-03-12T03:33:59.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:33:58.975+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3726 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:33:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:58.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:58.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:58.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:33:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:58.879+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:59.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:59.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:59.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:33:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:33:59.609+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:33:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:59.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:59.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:59.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:33:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:33:59.491+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:00.122 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:59.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:59.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:59.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:33:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:33:59.830+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:00.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:00.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:00.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:00.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:00.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:00.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:00.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:00.539+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:00.806 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:00.584+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:00.806 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:00.584+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:00.806 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:00.584+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:00.806 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:00.585+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:00.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:00.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:00.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:00.803+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:01.847 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:01.581+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:01.847 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:01.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:01.847 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:01.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:01.847 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:01.582+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:01.848 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:01.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:01.848 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:01.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:01.848 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:01.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:01.848 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:01.628+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:01.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:01.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:01.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:01.845+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:02.855 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:02.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:02.856 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:02.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:02.856 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:02.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:02.856 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:02.601+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:02.856 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:02.619+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:02.856 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:02.619+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:02.856 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:02.619+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:02.856 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:02.619+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:03.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.191+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:34:03.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.191+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:34:03.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:02.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:03.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:02.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:03.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:02.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:03.194 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:02.853+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:03.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:03.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:03.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:03.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:03.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:03.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:03.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:03.651+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:03.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:03.635+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:03.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:03.635+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:03.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:03.635+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:03.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:03.636+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003d370 tx=0x7fb7f004cd10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003d370 tx=0x7fb7f004cd10 comp rx=0 tx=0).stop 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:03.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb7e804f760 2024-03-12T03:34:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.975+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80501e0 con 0x7fb8000ffcc0 2024-03-12T03:34:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.976+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.976+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.976+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.976+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:34:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.976+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035040 tx=0x7fb7f001b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8052ca0 con 0x7fb7e804f760 2024-03-12T03:34:03.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804f760 2024-03-12T03:34:03.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:03.978+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8009850 con 0x7fb7e804f760 2024-03-12T03:34:04.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:03.976+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3788 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:03.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:03.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:03.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:04.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:03.867+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:04.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:04.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:04.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:04.828+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:04.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:04.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:04.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:04.656+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:04.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:04.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:04.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:04.621+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:05.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:05.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:05.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:05.628+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:05.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:05.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:05.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:05.820+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:05.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:05.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:05.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:05.639+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:06.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:06.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:06.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:06.681+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:06.771+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:06.771+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:06.771+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:06.772+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:06.666+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:06.666+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:06.666+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:06.666+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:07.682+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:07.682+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:07.682+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:07.683+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:07.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:07.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:07.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:07.750+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:07.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:07.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:07.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:08.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:07.642+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:08.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:08.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:08.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:08.649+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:08.696+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:08.696+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:08.696+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:08.697+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:08.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:08.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:08.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:08.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:08.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:08.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:08.742+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035040 tx=0x7fb7f001b470 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035040 tx=0x7fb7f001b470 comp rx=0 tx=0).stop 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb7e804fb40 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80501e0 con 0x7fb8000ffcc0 2024-03-12T03:34:08.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.976+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:08.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.977+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:08.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.977+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:08.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.977+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:34:08.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.977+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8017da0 tx=0x7fb7f8045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:08.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.978+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804fb40 2024-03-12T03:34:08.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.978+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8014570 con 0x7fb7e804fb40 2024-03-12T03:34:08.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:08.979+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8009850 con 0x7fb7e804fb40 2024-03-12T03:34:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:08.976+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3850 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:09.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:09.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:09.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:09.664+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:09.779+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:09.779+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:09.779+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:09.780+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:09.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:09.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:09.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:09.700+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:10.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:10.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:10.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:10.711+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:10.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:10.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:10.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:10.806+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:10.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:10.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:10.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:10.681+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:11.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:11.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:11.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:11.800+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:11.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:11.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:11.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:11.730+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:11.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:11.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:11.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:11.725+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:12.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:12.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:12.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:12.767+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:12.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:12.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:12.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:12.696+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:12.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:12.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:12.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:12.761+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:13.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:13.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:13.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:13.791+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:13.776+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:13.776+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:13.776+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:13.777+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:13.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:13.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:13.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:13.657+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8017da0 tx=0x7fb7f8045000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8017da0 tx=0x7fb7f8045000 comp rx=0 tx=0).stop 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb7e804f760 2024-03-12T03:34:13.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80501e0 con 0x7fb8000ffcc0 2024-03-12T03:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.977+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.978+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.978+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:13.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.978+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.978+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8007c10 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.979+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0031ef0 con 0x7fb7e804f760 2024-03-12T03:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.980+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0028220 con 0x7fb7e804f760 2024-03-12T03:34:13.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:13.980+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:34:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:13.978+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3913 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:14.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:14.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:14.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:14.647+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:14.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:14.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:14.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:14.783+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:14.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:14.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:14.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:14.783+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:15.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:15.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:15.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:15.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:15.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:15.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:15.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:15.641+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:15.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:15.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:15.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:15.772+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:15.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:16.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:15.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:16.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:15.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:16.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:15.760+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:16.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:16.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:16.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:16.785+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:16.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:16.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:16.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:16.597+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:16.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:16.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:16.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:17.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:16.792+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:17.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:17.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:17.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:17.762+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:17.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:17.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:17.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:17.769+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:17.583+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:17.583+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:17.583+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:17.584+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:18.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.551+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:34:18.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.551+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:34:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:18.798+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:18.798+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:18.798+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:18.799+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:18.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:18.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:18.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:18.573+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:18.779+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:18.779+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:18.779+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:18.780+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8007c10 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8007c10 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:18.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:34:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.978+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:34:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.979+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.979+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:18.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.979+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.979+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:34:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.980+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005f260 tx=0x7fb7f000b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000de60 con 0x7fb7e804fb40 2024-03-12T03:34:18.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f002d3f0 con 0x7fb7e804fb40 2024-03-12T03:34:18.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:18.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804fb40 2024-03-12T03:34:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:18.979+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 3974 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:19.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:19.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:19.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:19.539+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:19.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:19.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:19.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:19.813+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:19.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:19.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:19.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:19.811+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:20.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:20.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:20.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:20.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:20.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:20.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:20.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:20.492+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:20.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:20.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:20.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:20.840+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:20.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:20.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:20.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:20.801+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:21.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:21.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:21.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:21.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:21.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:21.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:21.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:21.485+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:21.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:21.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:21.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:21.818+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:21.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:21.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:21.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:21.785+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:22.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:22.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:22.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:22.532+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:22.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:22.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:22.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:22.841+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:22.823+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:22.823+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:22.823+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:22.824+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:23.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:23.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:23.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:23.783+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:23.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:23.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:23.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:23.536+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:23.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:34:23.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:23.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005f260 tx=0x7fb7f000b1b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb7f005f260 tx=0x7fb7f000b1b0 comp rx=0 tx=0).stop 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.979+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.980+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.980+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.980+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:23.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.980+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:34:23.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.980+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0022f80 tx=0x7fb7f800ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:23.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.980+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d7b0 con 0x7fb7e804f760 2024-03-12T03:34:23.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f002d3f0 con 0x7fb7e804f760 2024-03-12T03:34:23.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:23.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:34:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:23.980+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4036 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:23.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:23.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:23.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:23.847+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:24.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:24.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:24.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:24.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:24.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:24.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:24.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:24.965 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:24.754+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:24.966 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:24.754+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:24.966 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:24.754+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:24.966 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:24.755+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:24.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:24.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:24.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:24.828+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:25.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:25.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:25.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:25.551+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:25.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:25.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:25.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:25.796+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:25.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:25.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:25.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:25.740+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:26.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:26.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:26.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:26.540+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:26.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:26.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:26.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:26.769+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:26.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:26.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:26.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:26.712+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:27.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:27.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:27.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:27.767+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:27.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:27.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:27.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:27.581+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:27.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:27.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:27.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:27.678+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:28.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:28.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:28.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:28.711+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:28.611+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:28.611+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:28.611+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:28.612+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:28.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:28.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:28.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:28.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:28.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:28.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:28.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:28.769+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:28.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:34:28.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0022f80 tx=0x7fb7f800ddd0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0022f80 tx=0x7fb7f800ddd0 comp rx=0 tx=0).stop 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050260 con 0x7fb7e804fb40 2024-03-12T03:34:28.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.980+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb8000ffcc0 2024-03-12T03:34:28.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.981+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:28.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.981+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:28.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.981+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:28.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.981+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050b20 con 0x7fb7e804fb40 2024-03-12T03:34:28.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.981+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8012f90 tx=0x7fb7f0044390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:28.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804fb40 2024-03-12T03:34:28.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804fb40 2024-03-12T03:34:28.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:28.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8009850 con 0x7fb7e804fb40 2024-03-12T03:34:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:28.980+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4098 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:29.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:29.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:29.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:29.659+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:29.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:29.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:29.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:29.769+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:30.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:29.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:30.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:29.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:30.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:29.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:30.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:29.677+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:30.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:30.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:30.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:30.741+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:30.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:30.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:30.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:30.645+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:30.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:30.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:30.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:30.696+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:31.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:31.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:31.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:31.777+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:31.597+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:31.597+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:31.597+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:31.598+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:31.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:31.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:31.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:31.649+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:32.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:32.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:32.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:32.593+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:32.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:32.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:32.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:32.611+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:32.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:32.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:32.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:33.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:32.826+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:33.553+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:33.553+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:33.553+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:33.553+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:33.633+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:33.633+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:33.633+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:33.634+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:33.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.911+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:34:33.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.911+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:34:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:34:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8012f90 tx=0x7fb7f0044390 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8012f90 tx=0x7fb7f0044390 comp rx=0 tx=0).stop 2024-03-12T03:34:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:33.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:33.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:33.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050180 con 0x7fb7e804f760 2024-03-12T03:34:33.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80502c0 con 0x7fb8000ffcc0 2024-03-12T03:34:33.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.981+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:33.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.982+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:33.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.982+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:33.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.982+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:34:33.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.982+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8018010 tx=0x7fb7f804a860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:33.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d7b0 con 0x7fb7e804f760 2024-03-12T03:34:33.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000ee30 con 0x7fb7e804f760 2024-03-12T03:34:33.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:33.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071070 con 0x7fb7e804f760 2024-03-12T03:34:34.132 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:33.981+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4161 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:34.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:33.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:34.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:33.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:34.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:33.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:34.133 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:33.871+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:34.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:34.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:34.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:34.660+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:34.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:34.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:34.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:34.519+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:34.880+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:34.880+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:34.880+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:34.881+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:35.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:35.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:35.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:35.700+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:35.517+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:35.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:35.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:35.518+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:35.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:35.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:35.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:35.870+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:36.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:36.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:36.662+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:36.663+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:36.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:36.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:36.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:36.515+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:36.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:36.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:36.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:36.869+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:37.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:37.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:37.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:37.686+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:37.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:37.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:37.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:37.470+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:37.904+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:38.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:38.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:38.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:38.495+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:38.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:38.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:38.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:38.671+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:38.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.981+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:34:38.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.981+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.981+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8018010 tx=0x7fb7f804a860 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8018010 tx=0x7fb7f804a860 comp rx=0 tx=0).stop 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050180 con 0x7fb7e804fb40 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80502c0 con 0x7fb8000ffcc0 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.982+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.983+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.983+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:38.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.983+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:34:38.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.983+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00636d0 tx=0x7fb7f004b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:38.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f000d7b0 con 0x7fb7e804fb40 2024-03-12T03:34:38.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000ee30 con 0x7fb7e804fb40 2024-03-12T03:34:38.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:38.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:34:39.188 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:38.982+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4222 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:39.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:38.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:39.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:38.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:39.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:38.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:39.189 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:38.878+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:39.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:39.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:39.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:39.696+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:39.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:39.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:39.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:39.450+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:39.876+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:39.876+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:39.876+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:39.877+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:40.747 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:40.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:40.747 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:40.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:40.747 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:40.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:40.748 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:40.440+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:40.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:40.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:40.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:40.836+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:40.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:40.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:40.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:40.746+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:41.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:41.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:41.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:41.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:41.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:41.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:41.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:41.408+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:41.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:41.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:41.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:41.813+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:41.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:41.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:41.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:41.774+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:42.782 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:42.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:42.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:42.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:42.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:42.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:42.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:42.361+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:42.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:42.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:42.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:42.782+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:42.780+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:42.780+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:42.780+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:42.781+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:43.551 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:43.409+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:43.551 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:43.409+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:43.551 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:43.409+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:43.551 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:43.410+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:43.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:34:43.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:43.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00636d0 tx=0x7fb7f004b8c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00636d0 tx=0x7fb7f004b8c0 comp rx=0 tx=0).stop 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050180 con 0x7fb7e804f760 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.982+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80502c0 con 0x7fb8000ffcc0 2024-03-12T03:34:43.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.983+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:43.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.983+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:43.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.983+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:43.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.983+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:34:43.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.983+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0034f30 tx=0x7fb7f004b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:43.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045bd0 con 0x7fb7e804f760 2024-03-12T03:34:43.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00285c0 con 0x7fb7e804f760 2024-03-12T03:34:43.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:43.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804f760 2024-03-12T03:34:44.240 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:43.983+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4285 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:44.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:43.808+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:44.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:43.808+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:44.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:43.808+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:44.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:43.809+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:44.241 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:43.827+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:44.242 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:43.827+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:44.242 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:43.827+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:44.242 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:43.828+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:44.375+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:44.375+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:44.375+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:44.375+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:44.812+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:44.812+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:44.812+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:44.813+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:44.807+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:44.807+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:45.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:44.807+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:45.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:44.808+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:45.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:45.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:45.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:45.370+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:46.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:45.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:46.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:45.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:46.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:45.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:46.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:45.816+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:46.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:45.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:46.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:45.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:46.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:45.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:46.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:45.859+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:46.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:46.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:46.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:46.333+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:46.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:46.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:46.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:46.838+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:46.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:46.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:46.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:46.857+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:47.319+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:47.319+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:47.319+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:47.320+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:47.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:47.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:47.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:47.835+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:47.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:47.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:47.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:47.863+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:48.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:48.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:48.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:48.318+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:48.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:34:48.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:48.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0034f30 tx=0x7fb7f004b6b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:48.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0034f30 tx=0x7fb7f004b6b0 comp rx=0 tx=0).stop 2024-03-12T03:34:48.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:48.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:48.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:48.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:48.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050180 con 0x7fb7e804fb40 2024-03-12T03:34:48.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.983+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80502c0 con 0x7fb8000ffcc0 2024-03-12T03:34:48.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.984+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:48.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.984+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:48.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.984+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:48.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.984+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:34:48.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.984+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003d150 tx=0x7fb7f0038040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:48.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045bd0 con 0x7fb7e804fb40 2024-03-12T03:34:48.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00283f0 con 0x7fb7e804fb40 2024-03-12T03:34:48.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:48.986+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:34:49.283 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:48.983+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4348 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:49.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:48.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:49.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:48.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:49.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:48.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:49.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:48.871+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:49.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:48.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:49.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:48.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:49.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:48.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:49.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:48.825+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:49.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:49.335+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:34:49.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:49.335+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:34:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:49.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:49.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:49.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:49.281+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:49.847+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:49.847+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:49.847+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:49.848+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:49.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:49.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:49.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:49.796+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:50.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:50.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:50.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:50.232+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:50.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:50.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:50.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:50.766+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:50.860+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:50.860+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:50.860+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:50.861+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:51.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:51.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:51.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:51.205+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:51.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:51.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:51.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:51.746+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:51.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:51.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:51.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:51.877+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:52.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:52.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:52.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:52.211+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:52.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:52.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:52.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:52.736+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:52.828+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:52.828+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:53.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:52.828+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:53.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:52.829+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:53.551 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:53.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:53.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:53.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:53.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:53.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:53.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:53.181+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:53.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:53.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:53.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:53.775+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003d150 tx=0x7fb7f0038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb7f003d150 tx=0x7fb7f0038040 comp rx=0 tx=0).stop 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050180 con 0x7fb7e804f760 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80502c0 con 0x7fb8000ffcc0 2024-03-12T03:34:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.984+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.985+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.985+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.985+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:34:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.985+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8031490 tx=0x7fb7f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804f760 2024-03-12T03:34:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804f760 2024-03-12T03:34:53.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:53.987+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f804a470 con 0x7fb7e804f760 2024-03-12T03:34:54.233 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:53.984+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4412 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:54.233 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:53.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:54.234 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:53.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:54.234 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:53.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:54.234 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:53.814+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:54.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:54.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:54.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:54.232+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:54.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:54.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:54.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:54.793+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:54.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:54.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:54.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:54.810+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:55.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:55.571 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:55.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:55.571 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:55.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:55.571 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:55.266+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:55.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:55.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:55.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:55.811+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:55.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:56.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:55.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:56.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:55.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:56.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:55.788+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:56.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:56.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:56.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:56.238+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:56.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:56.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:56.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:56.843+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:56.757+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:56.757+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:56.757+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:56.757+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:57.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:57.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:57.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:57.219+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:57.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:57.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:57.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:57.879+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:57.716+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:57.716+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:57.716+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:57.717+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:58.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:58.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:58.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:58.267+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:58.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:34:58.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:34:58.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8031490 tx=0x7fb7f0008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8031490 tx=0x7fb7f0008040 comp rx=0 tx=0).stop 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.984+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:34:58.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.985+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:34:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.985+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:34:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.985+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:34:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.985+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:34:58.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.986+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8013340 tx=0x7fb7f8050b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:34:58.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.986+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804fb40 2024-03-12T03:34:58.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.986+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804fb40 2024-03-12T03:34:58.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:34:58.987+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f804a470 con 0x7fb7e804fb40 2024-03-12T03:34:59.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:34:58.985+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4474 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:34:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:58.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:58.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:58.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:34:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:58.925+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:34:59.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:58.747+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:59.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:58.747+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:59.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:58.747+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:34:59.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:58.748+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:34:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:59.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:59.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:59.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:34:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:34:59.236+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:59.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:59.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:59.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:34:59.719+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:59.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:59.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:59.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:34:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:34:59.936+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:00.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:00.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:00.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:00.218+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:00.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:00.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:00.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:00.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:00.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:00.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:00.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:00.677+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:00.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:00.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:00.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:00.980+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:01.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:01.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:01.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:01.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:01.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:01.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:01.213+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:01.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:01.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:01.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:01.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:01.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:01.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:01.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:01.630+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:02.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:02.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:02.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:02.189+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:01.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:01.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:01.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:01.946+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:02.922 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:02.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:02.922 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:02.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:02.922 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:02.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:02.923 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:02.599+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:03.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:02.919+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:03.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:02.919+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:03.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:02.919+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:03.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:02.920+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:03.552 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:03.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:03.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:03.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:03.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:03.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:03.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:03.184+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:03.617+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:03.617+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:03.617+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:03.618+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:03.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8013340 tx=0x7fb7f8050b50 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8013340 tx=0x7fb7f8050b50 comp rx=0 tx=0).stop 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.985+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:35:03.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.986+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:03.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.986+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:03.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.986+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:03.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.986+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:35:03.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.987+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8010fc0 tx=0x7fb7f800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:03.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.987+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804f760 2024-03-12T03:35:03.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.987+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804f760 2024-03-12T03:35:03.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:03.988+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f801f690 con 0x7fb7e804f760 2024-03-12T03:35:04.208 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:03.986+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4537 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:04.209 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:03.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:04.209 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:03.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:04.209 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:03.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:04.209 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:03.948+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:04.464 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:04.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:04.464 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:04.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:04.464 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:04.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:04.464 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:04.206+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:04.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:04.695+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:35:04.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:04.695+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:35:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:04.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:04.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:04.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:04.630+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:04.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:04.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:04.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:04.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:05.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:05.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:05.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:05.192+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:05.958 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:05.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:05.958 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:05.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:05.958 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:05.629+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:05.958 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:05.630+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:05.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:05.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:05.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:05.956+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:06.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:06.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:06.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:06.170+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:06.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:06.590+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:06.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:06.590+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:06.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:06.590+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:06.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:06.591+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:06.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:06.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:06.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:06.970+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:07.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:07.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:07.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:07.196+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:07.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:07.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:07.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:07.547+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:07.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:07.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:07.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:07.960+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:08.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:08.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:08.195+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:08.196+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:08.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:08.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:08.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:08.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8010fc0 tx=0x7fb7f800c9e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8010fc0 tx=0x7fb7f800c9e0 comp rx=0 tx=0).stop 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:08.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:35:08.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:35:08.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.986+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:08.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.987+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:08.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.987+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:08.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.987+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:35:08.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.987+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8040b70 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:08.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.989+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045bd0 con 0x7fb7e804fb40 2024-03-12T03:35:08.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.989+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0011c10 con 0x7fb7e804fb40 2024-03-12T03:35:08.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:08.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:35:09.228 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:08.987+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4600 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:09.228 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:09.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:09.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:09.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.007+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:09.521 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:09.225+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:09.521 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:09.225+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:09.521 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:09.225+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:09.521 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:09.226+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:09.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:09.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:09.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:09.583+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:10.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:10.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:10.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:10.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:09.993+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:10.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:10.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:10.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:10.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:10.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:10.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:10.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:10.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:10.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:10.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:10.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:10.541+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:11.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:10.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:11.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:10.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:11.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:10.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:11.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:10.946+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:11.239+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:11.239+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:11.239+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:11.239+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:11.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:11.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:11.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:11.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:11.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:11.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:11.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:11.587+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:12.237 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:11.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:12.237 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:11.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:12.238 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:11.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:12.238 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:11.984+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:12.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:12.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:12.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:12.235+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:13.018 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:12.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:13.018 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:12.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:13.018 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:12.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:13.018 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:12.568+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:13.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:13.015+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:13.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:13.015+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:13.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:13.015+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:13.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:13.015+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:13.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:13.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:13.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:13.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:13.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:13.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:13.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:13.272+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:13.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:13.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:13.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:13.556+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:13.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.987+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:13.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.987+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:13.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.987+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8040b70 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:13.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.987+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8040b70 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:35:13.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.987+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:13.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.987+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:13.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.987+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:13.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.988+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:13.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.988+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80500a0 con 0x7fb7e804f760 2024-03-12T03:35:13.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.988+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb8000ffcc0 2024-03-12T03:35:13.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.988+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:13.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.989+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:13.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.989+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:13.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.989+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050b20 con 0x7fb7e804f760 2024-03-12T03:35:13.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.989+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8031f00 tx=0x7fb7f0006c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:13.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801c2c0 con 0x7fb7e804f760 2024-03-12T03:35:13.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804f760 2024-03-12T03:35:13.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:13.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8057d50 con 0x7fb7e804f760 2024-03-12T03:35:14.260 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:13.988+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4665 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:14.260 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:14.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:14.260 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:14.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:14.260 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:14.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:14.260 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:14.065+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:14.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:14.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:14.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:14.258+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:14.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:14.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:14.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:14.560+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:15.559 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:15.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:15.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:15.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:15.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:15.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:15.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:15.296+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:15.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:15.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:15.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:15.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:15.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:15.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:15.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:15.113+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:15.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:15.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:15.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:15.557+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:16.127+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:16.127+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:16.127+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:16.127+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:16.303+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:16.303+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:16.303+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:16.304+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:16.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:16.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:16.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:16.602+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:17.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:17.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:17.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:17.131+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:17.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:17.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:17.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:17.342+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:17.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:17.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:17.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:17.647+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:18.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:18.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:18.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:18.175+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:18.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:18.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:18.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:18.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:18.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:18.312+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:18.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:18.313+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:18.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:18.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:18.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:18.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:18.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:18.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:18.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:18.687+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:18.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:35:18.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:18.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8031f00 tx=0x7fb7f0006c80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8031f00 tx=0x7fb7f0006c80 comp rx=0 tx=0).stop 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.988+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.989+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.989+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.989+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:18.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.989+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:35:18.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.989+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009790 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:18.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0011c10 con 0x7fb7e804fb40 2024-03-12T03:35:18.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000f8c0 con 0x7fb7e804fb40 2024-03-12T03:35:18.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:18.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:35:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:18.988+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4728 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:19.220+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:19.220+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:19.220+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:19.220+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:19.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:19.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:19.637 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:19.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:19.637 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:19.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:19.637 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:19.323+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:19.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:19.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:19.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:19.656+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:20.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:20.119+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:35:20.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:20.119+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:35:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:20.195+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:20.195+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:20.195+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:20.195+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:20.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:20.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:20.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:20.275+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:20.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:57.060158+0000 front 2024-03-12T03:24:57.060194+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:20.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:57.060213+0000 front 2024-03-12T03:24:57.060388+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:20.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:57.060360+0000 front 2024-03-12T03:24:57.060265+0000 (oldest deadline 2024-03-12T03:25:17.560090+0000) 2024-03-12T03:35:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:20.686+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:21.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:21.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:21.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:21.194+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:21.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:21.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:21.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:21.246+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:21.649+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:22.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:22.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:22.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:22.252+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:22.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:22.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:22.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:22.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:22.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:22.209+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:22.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:23.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:23.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:23.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:23.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:23.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:23.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:23.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:23.223+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:23.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:23.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:23.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:23.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:23.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:23.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:23.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:23.185+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:23.623+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:23.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:23.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:23.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009790 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:23.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009790 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:35:23.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:23.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.989+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.990+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.990+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.990+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.990+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d360 tx=0x7fb7f0024c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:23.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801c2c0 con 0x7fb7e804f760 2024-03-12T03:35:23.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804f760 2024-03-12T03:35:23.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:23.992+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8e0 con 0x7fb7e804f760 2024-03-12T03:35:24.258 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:23.989+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4792 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:24.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:24.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:24.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:24.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:24.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:24.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:24.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:24.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:24.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:24.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:24.256+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:24.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:24.666+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:25.290+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:25.290+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:25.290+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:25.291+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:25.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:25.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:25.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:25.197+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:25.639+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:26.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:26.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:26.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:26.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:26.158+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:26.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:26.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:26.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:26.319+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:26.685+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:27.371 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:27.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:27.371 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:27.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:27.371 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:27.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:27.371 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:27.110+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:27.647 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:27.368+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:27.647 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:27.368+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:27.647 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:27.368+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:27.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:27.369+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:27.645+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:28.406 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:28.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:28.406 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:28.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:28.406 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:28.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:28.406 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:28.138+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:28.637+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:28.403+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:28.403+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:28.403+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:28.403+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:28.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.989+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.989+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.989+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d360 tx=0x7fb7f0024c40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000d360 tx=0x7fb7f0024c40 comp rx=0 tx=0).stop 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:35:28.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.990+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:28.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.991+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:28.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.991+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:28.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.991+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:35:28.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.991+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800c010 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:28.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801c2c0 con 0x7fb7e804fb40 2024-03-12T03:35:28.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009420 con 0x7fb7e804fb40 2024-03-12T03:35:28.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:28.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8e0 con 0x7fb7e804fb40 2024-03-12T03:35:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:28.990+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4856 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:29.128+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.681557+0000 front 2024-03-12T03:24:56.681565+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:29.128+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.681483+0000 front 2024-03-12T03:24:56.681530+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:29.128+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.681543+0000 front 2024-03-12T03:24:56.681577+0000 (oldest deadline 2024-03-12T03:25:17.781448+0000) 2024-03-12T03:35:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:29.129+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:29.742 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:29.629+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:29.742 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:29.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:29.743 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:29.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:29.743 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:29.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:29.743 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:29.389+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:30.388 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:30.101+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:30.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:30.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:30.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:30.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:30.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:30.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:30.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:30.386+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:30.679+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:31.124+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:31.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:31.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:31.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:31.371+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:31.728+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:32.383 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:32.117+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:32.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:32.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:32.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:32.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:32.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:32.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:32.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:32.381+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:32.702+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:33.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:33.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:33.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:33.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:33.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:33.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:33.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:33.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:33.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:33.347+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:33.700+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:33.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:33.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800c010 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800c010 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.990+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.991+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.991+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.991+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.991+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:35:33.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.992+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80175f0 tx=0x7fb7f8023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:33.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0011c40 con 0x7fb7e804f760 2024-03-12T03:35:33.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0011c40 con 0x7fb7e804f760 2024-03-12T03:35:33.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:33.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804f760 2024-03-12T03:35:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:33.991+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4921 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:34.136+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:34.716 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:34.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:34.716 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:34.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:34.716 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:34.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:34.716 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:34.315+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:34.713+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:35.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:35.479+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:35:35.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:35.479+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:35:35.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:35.099+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:35.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:35.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 since back 2024-03-12T03:24:56.904936+0000 front 2024-03-12T03:24:56.904884+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:35.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:35.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 since back 2024-03-12T03:24:56.904863+0000 front 2024-03-12T03:24:56.904849+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:35.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:35.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 since back 2024-03-12T03:24:56.904896+0000 front 2024-03-12T03:24:56.904919+0000 (oldest deadline 2024-03-12T03:25:19.804796+0000) 2024-03-12T03:35:35.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:35.346+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:35.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:36.312+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:36.127+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:36.725+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:37.169+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:37.327+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:37.690+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:38.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:38.315+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:38.994 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:38.735+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80175f0 tx=0x7fb7f8023000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80175f0 tx=0x7fb7f8023000 comp rx=0 tx=0).stop 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:38.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.991+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.992+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.992+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.992+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.992+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003e30 tx=0x7fb7f004c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.992+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804fb40 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0011c40 con 0x7fb7e804fb40 2024-03-12T03:35:38.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:38.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804fb40 2024-03-12T03:35:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:38.991+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 4984 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:39.107+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:39.689 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:39.337+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:39.686+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:40.157+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:40.318+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:40.670+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:41.190+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:41.365+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:41.657+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:42.205+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:42.357+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:42.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:42.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:42.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:42.660+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:43.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:43.250+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:43.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:43.396+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:43.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:43.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:43.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:43.697+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003e30 tx=0x7fb7f004c3d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003e30 tx=0x7fb7f004c3d0 comp rx=0 tx=0).stop 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb7e804f760 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c60 con 0x7fb8000ffcc0 2024-03-12T03:35:43.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.992+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:43.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.993+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:43.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.993+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:43.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.993+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050da0 con 0x7fb7e804f760 2024-03-12T03:35:43.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.993+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803a040 tx=0x7fb7f0038040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:43.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801f470 con 0x7fb7e804f760 2024-03-12T03:35:43.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804f760 2024-03-12T03:35:43.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:43.995+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804f760 2024-03-12T03:35:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:43.992+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5048 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:44.210+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:44.685 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:44.380+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:44.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:44.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:44.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:44.683+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:45.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:45.162+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:45.661 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:45.383+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:45.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:45.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:45.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:45.660+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:46.170+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:46.401+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:46.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:46.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:46.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:46.642+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:47.138+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:47.354+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:47.636+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:47.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:47.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:47.637+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:48.120+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:48.332+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:48.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:48.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:48.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:48.642+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:48.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.992+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:48.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.992+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:48.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.992+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803a040 tx=0x7fb7f0038040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:48.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803a040 tx=0x7fb7f0038040 comp rx=0 tx=0).stop 2024-03-12T03:35:48.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:48.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:48.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:48.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:48.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:35:48.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050da0 con 0x7fb8000ffcc0 2024-03-12T03:35:48.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.993+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:48.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.994+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:48.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.994+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:48.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.994+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ee0 con 0x7fb7e804fb40 2024-03-12T03:35:48.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.994+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8043200 tx=0x7fb7f801df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:48.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.996+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801f470 con 0x7fb7e804fb40 2024-03-12T03:35:48.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.996+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8030eb0 con 0x7fb7e804fb40 2024-03-12T03:35:48.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:48.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804fb40 2024-03-12T03:35:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:48.993+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5111 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:49.132+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:49.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:49.345+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:49.966 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:49.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:49.966 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:49.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:49.966 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:49.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:49.966 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:49.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:50.309 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:50.118+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:50.305+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:50.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:50.839+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:35:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:50.839+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:35:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:50.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:50.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:50.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:50.643+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:51.084+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:51.332+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:51.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:51.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:51.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:51.628+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:52.339+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:52.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:52.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:52.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:52.119+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:52.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:52.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:52.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:52.597+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:53.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:53.322+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:53.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:53.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:53.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:53.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:53.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:53.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:53.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:53.153+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:53.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:53.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:53.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:53.581+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:53.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.993+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:35:53.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.993+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:53.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.993+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8043200 tx=0x7fb7f801df40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:53.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.993+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8043200 tx=0x7fb7f801df40 comp rx=0 tx=0).stop 2024-03-12T03:35:53.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:53.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.994+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:53.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.994+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:53.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.994+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:53.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050bd0 con 0x7fb7e804f760 2024-03-12T03:35:53.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d10 con 0x7fb8000ffcc0 2024-03-12T03:35:53.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.994+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:53.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.995+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:53.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.995+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:53.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.995+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e50 con 0x7fb7e804f760 2024-03-12T03:35:53.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.995+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8030780 tx=0x7fb7f8055c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:53.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.996+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804f760 2024-03-12T03:35:54.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0011c40 con 0x7fb7e804f760 2024-03-12T03:35:54.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:53.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804f760 2024-03-12T03:35:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:53.994+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5175 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:54.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:54.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:54.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:54.119+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:54.606 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:54.324+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:55.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:54.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:55.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:54.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:55.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:54.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:55.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:54.604+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:55.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:55.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:55.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:55.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:55.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:55.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:55.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:55.135+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:55.276+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:55.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:55.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:55.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:55.564+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:56.518 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:56.309+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:56.518 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:56.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:56.518 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:56.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:56.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:56.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:56.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:56.153+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:56.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:56.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:56.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:56.516+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:57.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:57.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:57.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:57.177+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:57.359+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:57.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:57.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:57.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:57.557+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:58.521 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:58.407+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:58.522 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:58.180+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:58.522 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:58.180+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:58.522 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:58.180+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:58.522 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:58.181+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:58.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:58.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:58.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:58.519+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:35:58.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8030780 tx=0x7fb7f8055c00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8030780 tx=0x7fb7f8055c00 comp rx=0 tx=0).stop 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050a70 con 0x7fb7e804fb40 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.994+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050bb0 con 0x7fb8000ffcc0 2024-03-12T03:35:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.995+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:35:58.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.995+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:35:58.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.995+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:35:58.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.995+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050cf0 con 0x7fb7e804fb40 2024-03-12T03:35:58.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.996+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035ff0 tx=0x7fb7f006da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:35:58.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801f470 con 0x7fb7e804fb40 2024-03-12T03:35:59.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800d970 con 0x7fb7e804fb40 2024-03-12T03:35:59.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:35:58.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804fb40 2024-03-12T03:35:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:35:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:35:58.995+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5237 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:35:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:59.221+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:59.221+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:59.221+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:35:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:35:59.222+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:35:59.361+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:35:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:59.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:59.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:59.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:35:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:35:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:35:59.530+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:00.399 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:00.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:00.399 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:00.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:00.399 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:00.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:00.400 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:00.254+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:00.396+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:00.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:00.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:00.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:00.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:01.281+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:01.281+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:01.281+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:01.282+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:01.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:01.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:01.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:01.376+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:01.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:01.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:01.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:01.522+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:02.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:02.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:02.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:02.348+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:02.499+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:02.499+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:02.499+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:02.500+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:02.278+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:02.278+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:02.278+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:02.278+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:03.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:03.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:03.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:03.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:03.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:03.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:03.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:03.390+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:03.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:03.236+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:03.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:03.236+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:03.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:03.236+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:03.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:03.236+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:03.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:03.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:03.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:03.522+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035ff0 tx=0x7fb7f006da30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035ff0 tx=0x7fb7f006da30 comp rx=0 tx=0).stop 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:36:03.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.995+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:36:03.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.996+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:03.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.996+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:03.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.996+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:03.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.996+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:36:04.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.997+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8016030 tx=0x7fb7f803e360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:04.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801f470 con 0x7fb7e804f760 2024-03-12T03:36:04.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8027e00 con 0x7fb7e804f760 2024-03-12T03:36:04.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:03.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804f760 2024-03-12T03:36:04.271 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:03.995+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5299 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:04.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:04.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:04.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:04.481+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:04.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:04.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:04.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:04.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:04.345+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:04.269+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:04.269+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:04.269+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:04.269+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:05.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:05.304+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:05.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:05.304+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:05.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:05.304+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:05.459 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:05.304+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:05.459 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:05.284+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:05.460 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:05.284+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:05.460 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:05.284+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:05.460 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:05.285+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:05.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:05.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:05.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:05.457+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:06.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:06.263+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:36:06.265 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:06.263+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:36:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:06.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:06.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:06.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:06.444+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:06.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:06.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:06.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:06.303+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:06.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:06.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:06.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:06.278+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:07.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:07.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:07.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:07.440+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:07.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:07.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:07.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:07.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:07.283+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:07.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:07.283+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:07.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:07.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:07.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:07.309+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:08.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:08.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:08.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:08.443+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:08.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:08.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:08.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:08.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:08.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:08.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:08.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:08.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:08.262+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:08.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:36:08.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:08.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8016030 tx=0x7fb7f803e360 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8016030 tx=0x7fb7f803e360 comp rx=0 tx=0).stop 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb7e804fb40 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c60 con 0x7fb8000ffcc0 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.996+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.997+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.997+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:09.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.997+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050da0 con 0x7fb7e804fb40 2024-03-12T03:36:09.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.997+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b5d0 tx=0x7fb7f802ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:09.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045890 con 0x7fb7e804fb40 2024-03-12T03:36:09.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a840 con 0x7fb7e804fb40 2024-03-12T03:36:09.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:08.999+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804fb40 2024-03-12T03:36:09.276 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:08.997+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5361 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:09.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:09.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:09.273+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:09.275+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:09.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:09.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:09.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:09.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:09.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:09.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:09.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:09.295+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:10.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:10.253+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:10.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:10.253+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:10.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:10.253+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:10.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:10.254+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:10.526 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:10.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:10.526 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:10.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:10.526 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:10.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:10.526 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:10.277+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:10.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:10.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:10.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:10.523+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:11.279+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:11.279+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:11.279+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:11.280+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:11.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:11.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:11.257+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:11.258+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:11.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:11.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:11.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:11.567+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:12.238+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:12.238+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:12.238+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:12.239+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:12.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:12.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:12.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:12.549 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:12.234+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:12.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:12.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:12.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:12.546+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:13.507 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:13.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:13.507 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:13.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:13.507 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:13.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:13.507 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:13.221+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:13.507 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:13.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:13.507 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:13.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:13.507 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:13.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:13.508 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:13.214+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:13.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:13.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:13.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:13.505+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b5d0 tx=0x7fb7f802ce30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b5d0 tx=0x7fb7f802ce30 comp rx=0 tx=0).stop 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:14.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb7e804f760 2024-03-12T03:36:14.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.997+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c60 con 0x7fb8000ffcc0 2024-03-12T03:36:14.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.998+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:14.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.998+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:14.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.998+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:14.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.998+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050da0 con 0x7fb7e804f760 2024-03-12T03:36:14.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.998+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006ec0 tx=0x7fb7f000b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:14.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.999+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0028ed0 con 0x7fb7e804f760 2024-03-12T03:36:14.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:13.999+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0028ed0 con 0x7fb7e804f760 2024-03-12T03:36:14.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:14.000+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0072d60 con 0x7fb7e804f760 2024-03-12T03:36:14.269 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:13.998+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5424 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:14.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:14.199+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:14.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:14.199+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:14.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:14.199+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:14.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:14.199+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:14.532 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:14.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:14.532 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:14.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:14.532 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:14.266+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:14.532 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:14.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:14.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:14.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:14.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:14.530+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:15.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:15.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:15.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:15.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:15.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:15.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:15.482 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:15.231+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:15.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:15.186+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:15.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:15.186+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:15.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:15.186+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:15.483 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:15.187+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:15.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:15.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:15.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:15.525+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:16.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:16.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:16.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:16.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:16.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:16.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:16.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:16.184+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:16.550 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:16.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:16.550 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:16.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:16.550 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:16.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:16.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:16.198+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:16.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:16.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:16.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:16.548+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:17.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:17.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:17.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:17.192+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:17.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:17.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:17.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:17.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:17.235+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:17.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:17.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:17.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:17.510+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:18.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:18.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:18.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:18.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:18.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:18.245+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:18.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:18.246+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:18.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:18.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:18.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:18.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:18.214+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:18.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:18.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:18.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:18.558+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:19.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006ec0 tx=0x7fb7f000b1b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006ec0 tx=0x7fb7f000b1b0 comp rx=0 tx=0).stop 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050a90 con 0x7fb7e804fb40 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050bd0 con 0x7fb8000ffcc0 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.998+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:19.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.999+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:19.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.999+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:19.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.999+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050d10 con 0x7fb7e804fb40 2024-03-12T03:36:19.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:18.999+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035040 tx=0x7fb7f0009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:19.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:19.000+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8027e00 con 0x7fb7e804fb40 2024-03-12T03:36:19.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:19.000+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804fb40 2024-03-12T03:36:19.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:19.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804fb40 2024-03-12T03:36:19.242 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:18.999+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5485 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:19.243 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:19.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:19.243 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:19.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:19.243 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:19.205+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:19.243 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:19.206+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:19.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:19.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:19.240+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:19.240+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:19.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:19.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:19.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:19.583+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:20.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:20.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:20.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:20.224+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:20.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:20.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:20.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:20.184+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:20.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:20.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:20.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:20.596+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:21.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:21.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:21.557 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:21.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:21.557 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:21.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:21.557 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:21.209+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:21.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:21.191+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:21.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:21.191+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:21.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:21.191+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:21.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:21.192+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:21.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:21.623+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:36:21.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:21.623+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:36:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:21.554+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:21.554+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:21.554+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:21.554+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:22.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:22.169+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:22.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:22.169+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:22.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:22.169+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:22.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:22.170+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:22.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:22.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:22.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:22.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:22.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:22.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:22.553 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:22.255+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:22.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:22.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:22.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:22.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:23.557 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:23.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:23.558 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:23.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:23.558 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:23.240+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:23.558 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:23.240+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:23.558 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:23.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:23.558 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:23.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:23.558 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:23.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:23.558 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:23.181+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:23.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:23.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:23.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:23.559+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035040 tx=0x7fb7f0009900 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8035040 tx=0x7fb7f0009900 comp rx=0 tx=0).stop 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050a90 con 0x7fb7e804f760 2024-03-12T03:36:24.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050bd0 con 0x7fb8000ffcc0 2024-03-12T03:36:24.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:23.999+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:24.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:24.000+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:24.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:24.000+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:24.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:24.000+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050d10 con 0x7fb7e804f760 2024-03-12T03:36:24.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:24.000+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f930 tx=0x7fb7f8027d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:24.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:24.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045890 con 0x7fb7e804f760 2024-03-12T03:36:24.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:24.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0028ed0 con 0x7fb7e804f760 2024-03-12T03:36:24.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:24.002+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0072d60 con 0x7fb7e804f760 2024-03-12T03:36:24.288 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:24.000+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5546 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:24.288 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:24.202+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:24.288 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:24.202+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:24.289 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:24.202+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:24.289 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:24.203+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:24.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:24.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:24.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:24.286+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:24.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:24.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:24.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:24.516+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:25.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:25.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:25.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:25.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:25.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:25.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:25.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:25.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:25.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:25.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:25.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:25.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:25.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:25.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:25.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:25.253+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:26.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:26.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:26.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:26.218+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:26.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:26.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:26.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:26.535 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:26.358+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:26.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:26.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:26.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:26.533+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:27.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:27.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:27.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:27.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:27.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:27.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:27.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:27.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:27.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:27.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:27.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:27.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:27.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:27.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:27.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:27.240+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:27.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:27.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:27.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:27.554+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:28.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:28.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:28.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:28.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:28.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:28.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:28.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:28.385+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:28.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:28.204+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:28.548 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:28.204+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:28.548 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:28.204+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:28.548 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:28.204+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:28.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:28.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:28.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:28.545+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f930 tx=0x7fb7f8027d90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f930 tx=0x7fb7f8027d90 comp rx=0 tx=0).stop 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b40 con 0x7fb7e804fb40 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.000+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c80 con 0x7fb8000ffcc0 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.001+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.001+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:29.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.001+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:29.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.001+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050dc0 con 0x7fb7e804fb40 2024-03-12T03:36:29.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.002+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0019720 tx=0x7fb7f0047070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:29.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.002+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045890 con 0x7fb7e804fb40 2024-03-12T03:36:29.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.002+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0028ed0 con 0x7fb7e804fb40 2024-03-12T03:36:29.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:29.004+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0072d60 con 0x7fb7e804fb40 2024-03-12T03:36:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:29.001+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5608 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:29.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:29.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:29.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:29.160+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:29.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:29.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:29.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:29.504+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:29.431+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:29.431+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:29.431+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:29.432+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:30.416 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:30.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:30.416 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:30.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:30.416 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:30.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:30.416 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:30.384+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:30.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:30.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:30.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:30.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:30.417 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:30.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:30.417 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:30.154+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:30.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:30.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:30.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:30.456+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:31.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:31.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:31.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:31.163+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:31.409+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:31.409+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:31.409+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:31.504 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:31.409+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:31.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:31.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:31.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:31.502+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:32.524 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:32.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:32.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:32.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:32.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:32.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:32.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:32.184+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:32.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:32.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:32.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:32.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:32.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:32.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:32.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:32.390+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:32.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:32.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:32.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:32.523+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:33.440 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:33.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:33.441 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:33.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:33.441 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:33.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:33.441 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:33.164+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:33.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:33.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:33.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:33.478+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:33.438+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:33.438+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:33.438+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:33.438+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0019720 tx=0x7fb7f0047070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0019720 tx=0x7fb7f0047070 comp rx=0 tx=0).stop 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:34.004 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:36:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:36:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.002+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.002+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.002+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.002+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:36:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.002+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8042010 tx=0x7fb7f00386d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:34.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.004+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8027e00 con 0x7fb7e804f760 2024-03-12T03:36:34.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.004+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f804a4c0 con 0x7fb7e804f760 2024-03-12T03:36:34.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:34.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804f760 2024-03-12T03:36:34.176 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:34.002+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5670 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:34.457 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:34.173+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:34.457 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:34.173+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:34.457 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:34.173+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:34.457 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:34.174+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:34.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:34.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:34.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:34.454+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:34.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:34.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:34.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:34.494+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:35.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:35.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:35.474 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:35.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:35.475 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:35.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:35.475 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:35.177+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:35.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:35.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:35.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:35.498+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:35.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:35.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:35.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:35.538+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:36.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:36.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:36.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:36.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:36.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:36.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:36.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:36.139+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:36.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:36.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:36.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:36.573+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:36.536+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:36.536+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:36.536+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:36.536+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:37.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:37.111+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:36:37.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:37.111+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:36:37.502 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:37.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:37.502 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:37.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:37.502 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:37.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:37.502 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:37.185+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:37.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:37.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:37.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:37.544+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:37.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:37.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:37.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:37.500+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:38.508 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:38.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:38.508 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:38.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:38.508 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:38.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:38.509 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:38.216+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:38.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:38.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:38.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:38.506+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:38.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:38.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:38.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:38.505+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:39.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:36:39.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:39.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8042010 tx=0x7fb7f00386d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:39.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8042010 tx=0x7fb7f00386d0 comp rx=0 tx=0).stop 2024-03-12T03:36:39.005 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:39.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:39.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:39.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:39.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:36:39.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.001+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:36:39.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.002+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:39.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.002+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:39.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.002+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:39.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.002+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:36:39.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.002+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802cb40 tx=0x7fb7f804b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:39.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.003+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8027e00 con 0x7fb7e804fb40 2024-03-12T03:36:39.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.003+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f804a4c0 con 0x7fb7e804fb40 2024-03-12T03:36:39.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:39.004+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804fb40 2024-03-12T03:36:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:39.001+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5732 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:39.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:39.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:39.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:39.176+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:39.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:39.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:39.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:39.530+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:39.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:39.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:39.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:39.526+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:40.520 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:40.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:40.520 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:40.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:40.520 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:40.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:40.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:40.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:40.517+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:40.517+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:40.517+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:40.518+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:40.545+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:40.545+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:40.545+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:40.545+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:41.506 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:41.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:41.506 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:41.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:41.507 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:41.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:41.507 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:41.186+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:41.503+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:41.503+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:41.503+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:41.505+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:41.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:41.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:41.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:41.596+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:42.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:42.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:42.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:42.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:42.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:42.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:42.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:42.201+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:42.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:42.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:42.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:42.600+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:42.481+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:42.481+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:42.481+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:42.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:42.482+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:43.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:43.237+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:43.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:43.237+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:43.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:43.237+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:43.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:43.238+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:43.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:43.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:43.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:43.495+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:43.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:43.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:43.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:43.566+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:44.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:36:44.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802cb40 tx=0x7fb7f804b610 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802cb40 tx=0x7fb7f804b610 comp rx=0 tx=0).stop 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.003+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.004+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.004+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.004+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:44.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.004+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:36:44.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.005+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8007c10 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:44.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801f470 con 0x7fb7e804f760 2024-03-12T03:36:44.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800c710 con 0x7fb7e804f760 2024-03-12T03:36:44.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:44.006+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804f760 2024-03-12T03:36:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:44.003+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5793 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:44.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:44.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:44.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:44.208+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:44.611 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:44.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:44.611 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:44.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:44.611 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:44.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:44.611 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:44.449+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:44.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:44.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:44.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:44.609+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:45.231+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:45.231+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:45.231+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:45.232+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:45.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:45.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:45.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:45.454+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:45.563+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:45.563+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:45.563+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:45.564+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:46.444 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:46.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:46.444 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:46.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:46.445 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:46.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:46.445 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:46.187+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:46.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:46.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:46.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:46.530+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:46.442+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:46.442+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:46.442+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:46.443+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:47.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:47.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:47.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:47.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:47.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:47.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:47.525 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:47.463+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:47.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:47.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:47.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:47.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:47.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:47.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:47.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:47.223+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:47.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:47.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:47.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:47.522+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:48.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:48.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:48.526 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:48.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:48.526 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:48.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:48.526 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:48.184+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:48.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:48.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:48.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:48.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:48.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:48.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:48.526 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:48.415+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:48.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:48.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:48.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:48.523+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8007c10 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:49.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8007c10 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.004+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:36:49.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.005+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dcc0 tx=0x7fb7f8047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045890 con 0x7fb7e804fb40 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0067400 con 0x7fb7e804fb40 2024-03-12T03:36:49.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:49.006+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000dcc0 con 0x7fb7e804fb40 2024-03-12T03:36:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:49.004+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5856 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:49.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:49.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:49.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:49.188+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:49.506+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:49.506+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:49.506+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:49.507+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:49.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:49.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:49.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:49.443+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:50.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:50.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:50.450 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:50.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:50.450 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:50.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:50.450 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:50.184+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:50.447+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:50.447+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:50.447+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:50.448+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:50.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:50.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:50.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:50.544+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:51.464 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:51.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:51.464 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:51.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:51.464 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:51.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:51.464 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:51.188+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:51.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:51.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:51.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:51.462+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:51.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:51.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:51.494+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:51.495+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:52.419 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:52.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:52.419 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:52.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:52.419 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:52.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:52.420 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:52.160+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:52.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:52.471+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:36:52.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:52.471+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:36:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:52.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:52.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:52.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:52.446+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:52.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:52.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:52.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:52.417+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:53.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:53.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:53.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:53.379+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:53.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:53.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:53.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:53.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:53.177+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:53.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:53.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:53.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:53.447+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:54.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.004+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.004+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.004+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dcc0 tx=0x7fb7f8047040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800dcc0 tx=0x7fb7f8047040 comp rx=0 tx=0).stop 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804f760 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:36:54.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:54.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.005+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:54.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.006+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:54.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.006+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804f760 2024-03-12T03:36:54.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.007+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046920 tx=0x7fb7f00623a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:54.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.007+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801f470 con 0x7fb7e804f760 2024-03-12T03:36:54.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.007+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009460 con 0x7fb7e804f760 2024-03-12T03:36:54.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:54.008+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f804aba0 con 0x7fb7e804f760 2024-03-12T03:36:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:54.005+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5918 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:54.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:54.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:54.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:54.133+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:54.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:54.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:54.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:54.364+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:54.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:54.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:54.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:54.438+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:55.429 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:55.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:55.429 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:55.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:55.429 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:55.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:55.429 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:55.152+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:55.429 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:55.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:55.430 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:55.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:55.430 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:55.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:55.430 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:55.350+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:55.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:55.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:55.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:55.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:55.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:55.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:55.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:55.427+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:56.391 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:56.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:56.391 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:56.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:56.391 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:56.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:56.391 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:56.133+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:56.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:56.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:56.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:56.460+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:56.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:56.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:56.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:56.389+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:57.513 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:57.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:57.513 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:57.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:57.513 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:57.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:57.513 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:57.345+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:57.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:57.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:57.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:57.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:57.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:57.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:57.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:57.132+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:57.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:57.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:57.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:57.511+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:58.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:58.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:58.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:58.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:58.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:58.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:58.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:58.341+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:58.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:58.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:58.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:58.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:58.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:58.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:58.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:58.099+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:58.552+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:58.552+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:58.552+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:58.553+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:36:59.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:36:59.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:36:59.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046920 tx=0x7fb7f00623a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:36:59.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046920 tx=0x7fb7f00623a0 comp rx=0 tx=0).stop 2024-03-12T03:36:59.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80504a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:36:59.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80504a0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:59.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:59.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:36:59.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509e0 con 0x7fb7e804fb40 2024-03-12T03:36:59.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b20 con 0x7fb8000ffcc0 2024-03-12T03:36:59.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.006+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:36:59.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.007+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:36:59.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.007+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:36:59.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.007+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c60 con 0x7fb7e804fb40 2024-03-12T03:36:59.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.007+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804ca90 tx=0x7fb7f803fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:36:59.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045890 con 0x7fb7e804fb40 2024-03-12T03:36:59.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f004bce0 con 0x7fb7e804fb40 2024-03-12T03:36:59.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:36:59.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000dcc0 con 0x7fb7e804fb40 2024-03-12T03:36:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:36:59.006+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 5980 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:36:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:59.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:59.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:59.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:36:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:36:59.085+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:59.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:59.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:59.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:36:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:36:59.368+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:36:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:59.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:59.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:59.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:36:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:36:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:36:59.602+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:00.328 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:00.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:00.328 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:00.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:00.328 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:00.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:00.328 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:00.066+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:00.637 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:00.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:00.637 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:00.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:00.637 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:00.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:00.637 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:00.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:01.045 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:00.634+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:01.045 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:00.634+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:01.046 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:00.634+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:01.046 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:00.635+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:01.042+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:01.042+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:01.042+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:01.043+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:01.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:01.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:01.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:01.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:01.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:01.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:01.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:01.332+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:02.009 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:01.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:02.009 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:01.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:02.009 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:01.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:02.010 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:01.616+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:02.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:02.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:02.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:02.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:02.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:02.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:02.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:02.007+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:02.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:02.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:02.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:02.301+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:03.054 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:02.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:03.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:02.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:03.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:02.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:03.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:02.616+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:03.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:03.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:03.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:03.259+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:03.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:03.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:03.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:03.052+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:03.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:03.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:03.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:03.590+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.006+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:37:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804ca90 tx=0x7fb7f803fb60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804ca90 tx=0x7fb7f803fb60 comp rx=0 tx=0).stop 2024-03-12T03:37:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e8050480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:04.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:04.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509c0 con 0x7fb7e804f760 2024-03-12T03:37:04.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b00 con 0x7fb8000ffcc0 2024-03-12T03:37:04.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.007+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e8050480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.008+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.008+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.009+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c40 con 0x7fb7e804f760 2024-03-12T03:37:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.009+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e8050480 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b870 tx=0x7fb7f001bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045890 con 0x7fb7e804f760 2024-03-12T03:37:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0035eb0 con 0x7fb7e804f760 2024-03-12T03:37:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:04.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000dcc0 con 0x7fb7e804f760 2024-03-12T03:37:04.304 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:04.007+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6041 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:04.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:04.094+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:04.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:04.094+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:04.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:04.094+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:04.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:04.095+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:04.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:04.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:04.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:04.303+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:04.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:04.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:04.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:04.607+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:05.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:05.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:05.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:05.110+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:05.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:05.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:05.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:05.296+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:05.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:05.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:05.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:05.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:06.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:06.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:06.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:06.275+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:06.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:06.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:06.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:06.107+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:06.669+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:06.669+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:06.669+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:06.670+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:07.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:07.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:07.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:07.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:07.227+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:07.227+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:07.227+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:07.228+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:07.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:07.831+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:37:07.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:07.831+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:37:08.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:07.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:08.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:07.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:08.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:07.641+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:08.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:07.641+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:08.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:08.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:08.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:08.056+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:08.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:08.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:08.246+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:08.247+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:08.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:08.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:08.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:08.656+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:09.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e8050480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:09.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e8050480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:09.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e8050480 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b870 tx=0x7fb7f001bd60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e8050480 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fb7f004b870 tx=0x7fb7f001bd60 comp rx=0 tx=0).stop 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e8050480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e8050480 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80509c0 con 0x7fb7e804fb40 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.008+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b00 con 0x7fb8000ffcc0 2024-03-12T03:37:09.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.009+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:09.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.009+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:09.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.009+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:09.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.009+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050c40 con 0x7fb7e804fb40 2024-03-12T03:37:09.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.010+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8039000 tx=0x7fb7f0036c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:09.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.011+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb80 con 0x7fb7e804fb40 2024-03-12T03:37:09.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.011+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800c710 con 0x7fb7e804fb40 2024-03-12T03:37:09.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:09.013+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804fb40 2024-03-12T03:37:09.249 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:09.008+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6103 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:09.250 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:09.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:09.250 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:09.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:09.250 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:09.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:09.250 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:09.025+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:09.247+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:09.247+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:09.247+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:09.247+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:10.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:09.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:10.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:09.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:10.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:09.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:10.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:09.647+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:10.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:10.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:10.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:10.207+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:10.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:10.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:10.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:10.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:10.046+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:10.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:10.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:10.695+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:10.696+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:11.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:11.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:11.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:11.073+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:11.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:11.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:11.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:11.232+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:11.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:11.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:11.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:11.714+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:12.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:12.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:12.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:12.281+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:12.075+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:12.075+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:12.075+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:12.076+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:12.709+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:12.709+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:12.709+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:12.710+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:13.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:13.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:13.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:13.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:13.242+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:13.242+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:13.242+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:13.242+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:13.691+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:13.691+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:13.691+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:13.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:13.692+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:14.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.008+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8039000 tx=0x7fb7f0036c20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8039000 tx=0x7fb7f0036c20 comp rx=0 tx=0).stop 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050920 con 0x7fb7e804f760 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050a60 con 0x7fb8000ffcc0 2024-03-12T03:37:14.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.009+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:14.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.010+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:14.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.010+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:14.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.010+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ba0 con 0x7fb7e804f760 2024-03-12T03:37:14.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.010+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017c10 tx=0x7fb7f8015510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:14.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb80 con 0x7fb7e804f760 2024-03-12T03:37:14.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.011+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f804aa20 con 0x7fb7e804f760 2024-03-12T03:37:14.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:14.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804f760 2024-03-12T03:37:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:14.009+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6166 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:14.209+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:14.209+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:14.209+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:14.209+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:14.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:14.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:14.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:14.093+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:15.056 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:14.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:15.056 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:14.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:15.056 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:14.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:15.056 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:14.682+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:15.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:15.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:15.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:15.204+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:15.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:15.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:15.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:15.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:15.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:15.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:15.054+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:16.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:15.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:16.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:15.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:16.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:15.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:16.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:15.683+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:16.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:16.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:16.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:16.056+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:16.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:16.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:16.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:16.244+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:17.020 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:16.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:17.020 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:16.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:17.021 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:16.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:17.021 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:16.692+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.018+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:17.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:17.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:17.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:17.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:17.250+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:17.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:17.650+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:17.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:17.650+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:17.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:17.650+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:17.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:17.650+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:18.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:18.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:18.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:18.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:17.980+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:18.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:18.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:18.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:18.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:18.237+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:18.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:18.652+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:18.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:18.652+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:18.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:18.652+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:18.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:18.652+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.009+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.009+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.009+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017c10 tx=0x7fb7f8015510 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.009+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017c10 tx=0x7fb7f8015510 comp rx=0 tx=0).stop 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.009+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050920 con 0x7fb7e804fb40 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050a60 con 0x7fb8000ffcc0 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.010+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.011+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:19.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.011+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:19.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.011+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ba0 con 0x7fb7e804fb40 2024-03-12T03:37:19.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.011+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046d90 tx=0x7fb7f8050980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:19.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045890 con 0x7fb7e804fb40 2024-03-12T03:37:19.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a8f0 con 0x7fb7e804fb40 2024-03-12T03:37:19.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:19.013+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f000dcc0 con 0x7fb7e804fb40 2024-03-12T03:37:19.231 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:19.010+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6229 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:19.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:18.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:19.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:18.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:19.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:18.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:19.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:18.970+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:19.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:19.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:19.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:19.230+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:19.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:19.694+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:19.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:19.694+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:19.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:19.694+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:19.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:19.694+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:20.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:19.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:20.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:19.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:20.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:19.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:20.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:19.984+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:20.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:20.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:20.249+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:20.250+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:20.937 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:20.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:20.938 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:20.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:20.938 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:20.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:20.938 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:20.681+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:21.301 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:20.935+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:21.301 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:20.935+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:21.301 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:20.935+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:21.301 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:20.936+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:21.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:21.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:21.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:21.299+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:21.905+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:21.905+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:21.905+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:21.906+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:21.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:22.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:21.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:22.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:21.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:22.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:21.701+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:22.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:22.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:22.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:22.297+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:22.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:22.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:22.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:22.907+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:22.749+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:22.749+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:22.749+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:22.749+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:23.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:23.255+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:37:23.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:23.255+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:37:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:23.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:23.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:23.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:23.288+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:23.871 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:23.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:23.871 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:23.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:23.871 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:23.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:23.871 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:23.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:23.871 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:23.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:23.872 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:23.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:23.872 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:23.796+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:24.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046d90 tx=0x7fb7f8050980 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8046d90 tx=0x7fb7f8050980 comp rx=0 tx=0).stop 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050920 con 0x7fb7e804f760 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050a60 con 0x7fb8000ffcc0 2024-03-12T03:37:24.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.011+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:24.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.012+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:24.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.012+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:24.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.012+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ba0 con 0x7fb7e804f760 2024-03-12T03:37:24.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.012+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8025f90 tx=0x7fb7f0045820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:24.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.013+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009470 con 0x7fb7e804f760 2024-03-12T03:37:24.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.014+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804f760 2024-03-12T03:37:24.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:24.014+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804f760 2024-03-12T03:37:24.287 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:24.011+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6287 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:24.288 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:23.869+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:24.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:24.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:24.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:24.285+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:24.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:24.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:24.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:24.891+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:24.806+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:24.806+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:24.806+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:24.807+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:25.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:25.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:25.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:25.297+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:26.257 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:25.817+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:26.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:25.817+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:26.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:25.817+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:26.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:25.818+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:26.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:25.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:26.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:25.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:26.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:25.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:26.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:25.862+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:26.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:26.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:26.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:26.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:27.290 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:26.854+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:27.290 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:26.854+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:27.291 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:26.854+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:27.291 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:26.854+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:27.291 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:26.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:27.291 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:26.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:27.291 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:26.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:27.291 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:26.891+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:27.287+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:27.287+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:27.287+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:27.289+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:27.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:27.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:27.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:27.813+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:27.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:27.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:28.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:27.890+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:28.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:27.891+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:28.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:28.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:28.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:28.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:28.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:28.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:28.771 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:28.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:29.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:29.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8025f90 tx=0x7fb7f0045820 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8025f90 tx=0x7fb7f0045820 comp rx=0 tx=0).stop 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050a60 con 0x7fb7e804fb40 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.010+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ba0 con 0x7fb8000ffcc0 2024-03-12T03:37:29.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.011+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:29.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.011+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:29.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.011+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:29.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.011+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ce0 con 0x7fb7e804fb40 2024-03-12T03:37:29.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.012+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00108a0 tx=0x7fb7f8031ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:29.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f003a760 con 0x7fb7e804fb40 2024-03-12T03:37:29.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0044980 con 0x7fb7e804fb40 2024-03-12T03:37:29.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:29.014+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:37:29.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:29.011+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6347 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:28.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:28.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:28.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:28.767+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:29.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:28.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:29.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:28.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:29.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:28.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:29.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:28.883+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:29.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:29.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:29.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:29.297+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:29.731+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:29.731+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:29.731+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:29.732+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:29.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:29.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:29.835+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:29.836+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:30.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:30.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:30.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:30.272+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:30.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:30.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:30.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:30.866+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:30.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:30.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:30.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:30.737+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:31.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:31.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:31.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:31.308+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:31.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:31.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:31.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:31.778+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:32.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:31.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:32.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:31.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:32.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:31.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:32.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:31.909+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:32.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:32.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:32.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:32.298+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:32.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:32.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:32.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:33.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:32.752+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:32.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:33.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:32.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:33.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:32.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:33.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:32.929+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:33.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:33.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:33.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:33.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:33.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:33.306+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:33.928 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:33.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:33.928 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:33.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:33.928 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:33.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:33.928 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:33.752+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00108a0 tx=0x7fb7f8031ec0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00108a0 tx=0x7fb7f8031ec0 comp rx=0 tx=0).stop 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:34.015 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ba0 con 0x7fb7e804f760 2024-03-12T03:37:34.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb8000ffcc0 2024-03-12T03:37:34.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.012+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:34.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.013+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:34.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.013+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:34.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.013+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e20 con 0x7fb7e804f760 2024-03-12T03:37:34.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.013+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0033be0 tx=0x7fb7f0023d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:34.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0011c10 con 0x7fb7e804f760 2024-03-12T03:37:34.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f00437c0 con 0x7fb7e804f760 2024-03-12T03:37:34.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:34.017+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804f760 2024-03-12T03:37:34.309 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:34.013+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6404 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:34.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:33.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:34.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:33.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:34.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:33.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:34.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:33.926+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:34.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:34.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:34.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:34.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:34.307+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:34.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:34.308+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:34.933+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:34.933+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:34.933+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:34.934+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:34.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:34.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:34.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:34.728+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:35.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:35.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:35.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:35.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:35.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:35.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:35.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:35.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:35.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:35.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:35.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:35.738+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:36.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:35.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:36.282 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:35.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:36.282 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:35.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:36.282 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:35.954+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:36.279+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:36.279+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:36.279+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:36.280+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:36.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:36.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:36.964+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:36.965+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:36.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:36.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:36.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:36.736+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:37.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:37.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:37.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:37.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:37.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:37.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:37.776 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:37.318+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:37.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:37.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:37.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:37.932+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:37.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:37.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:37.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:37.774+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:38.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:38.615+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:37:38.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:38.615+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:37:38.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:38.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:38.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:38.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:38.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:38.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:38.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:38.312+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:38.913 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:38.784+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:38.913 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:38.784+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:38.913 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:38.784+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:38.913 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:38.785+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0033be0 tx=0x7fb7f0023d90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0033be0 tx=0x7fb7f0023d90 comp rx=0 tx=0).stop 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:39.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:39.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ba0 con 0x7fb7e804fb40 2024-03-12T03:37:39.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.013+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb8000ffcc0 2024-03-12T03:37:39.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.014+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:39.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.014+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:39.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.014+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:39.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.014+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e20 con 0x7fb7e804fb40 2024-03-12T03:37:39.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.014+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006d70 tx=0x7fb7f000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:39.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.015+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0011c10 con 0x7fb7e804fb40 2024-03-12T03:37:39.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.015+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a930 con 0x7fb7e804fb40 2024-03-12T03:37:39.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:39.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071070 con 0x7fb7e804fb40 2024-03-12T03:37:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:39.014+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6458 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:38.910+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:38.910+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:38.910+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:38.911+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:39.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:39.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:39.802 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:39.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:39.802 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:39.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:39.802 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:39.319+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:39.943+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:39.943+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:39.943+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:39.944+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:39.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:39.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:39.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:39.800+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:40.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:40.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:40.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:40.297+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:40.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:40.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:40.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:40.984+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:40.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:40.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:40.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:40.796+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:41.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:41.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:41.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:41.263+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:41.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:41.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:41.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:41.954+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:41.779+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:41.780+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:41.780+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:41.780+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:42.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:42.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:42.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:42.233+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:43.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:42.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:43.236 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:42.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:43.236 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:42.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:43.236 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:42.984+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:43.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:42.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:43.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:42.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:43.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:42.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:43.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:42.815+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:43.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:43.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:43.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:43.234+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:43.938 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:43.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:43.938 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:43.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:43.938 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:43.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:43.938 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:43.793+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:44.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:44.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:44.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006d70 tx=0x7fb7f000b890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:44.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0006d70 tx=0x7fb7f000b890 comp rx=0 tx=0).stop 2024-03-12T03:37:44.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:44.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:44.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:44.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:44.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c50 con 0x7fb7e804f760 2024-03-12T03:37:44.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.014+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d90 con 0x7fb8000ffcc0 2024-03-12T03:37:44.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.015+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:44.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.015+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:44.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.015+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:44.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.015+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ed0 con 0x7fb7e804f760 2024-03-12T03:37:44.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.015+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0043c90 tx=0x7fb7f0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:44.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0011c10 con 0x7fb7e804f760 2024-03-12T03:37:44.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f003a930 con 0x7fb7e804f760 2024-03-12T03:37:44.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:44.017+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071070 con 0x7fb7e804f760 2024-03-12T03:37:44.195 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:44.015+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6513 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:44.195 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:43.935+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:44.195 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:43.935+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:44.195 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:43.935+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:44.195 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:43.936+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:44.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:44.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:44.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:44.193+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:44.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:44.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:44.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:44.924+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:44.804+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:44.804+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:45.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:44.804+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:45.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:44.805+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:45.146+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:45.146+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:45.146+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:45.148+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:45.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:45.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:45.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:45.811+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:45.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:45.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:45.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:46.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:45.908+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:46.129+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:46.129+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:46.129+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:46.130+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:46.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:46.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:46.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:46.770+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:46.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:47.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:46.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:47.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:46.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:47.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:46.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:47.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:47.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:47.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:47.109+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:47.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:47.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:47.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:47.752+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:48.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:47.942+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:48.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:47.942+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:48.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:47.942+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:48.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:47.942+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:48.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:48.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:48.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:48.125+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:48.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:48.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:48.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:48.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:48.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:48.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:48.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:48.712+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:48.965 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:48.961+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:48.965 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:48.961+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:48.965 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:48.961+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0043c90 tx=0x7fb7f0008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0043c90 tx=0x7fb7f0008040 comp rx=0 tx=0).stop 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:49.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:49.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb7e804fb40 2024-03-12T03:37:49.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e20 con 0x7fb8000ffcc0 2024-03-12T03:37:49.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.015+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:49.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.016+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:49.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.016+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:49.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.016+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f60 con 0x7fb7e804fb40 2024-03-12T03:37:49.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.016+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8038f20 tx=0x7fb7f0045890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:49.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.017+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009470 con 0x7fb7e804fb40 2024-03-12T03:37:49.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.017+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804fb40 2024-03-12T03:37:49.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:49.018+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f800c8c0 con 0x7fb7e804fb40 2024-03-12T03:37:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:49.015+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6570 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:48.962+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:49.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:49.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:49.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:49.103+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:49.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:49.705+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:49.705+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:49.705+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:49.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:49.706+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:49.973+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:49.973+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:49.973+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:49.973+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:50.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:50.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:50.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:50.139+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:50.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:50.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:50.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:50.959+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:50.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:50.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:50.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:50.720+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:51.329 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:51.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:51.329 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:51.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:51.329 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:51.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:51.329 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:51.110+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:51.963+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:51.963+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:51.963+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:51.965+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:51.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:51.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:51.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:51.767+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:52.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:52.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:52.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:52.072+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:52.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:52.774+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:52.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:52.774+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:52.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:52.774+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:52.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:52.775+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:53.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:53.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:53.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:53.043+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:52.990+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:52.990+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:52.990+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:53.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:52.991+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:53.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:53.975+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:37:53.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:53.975+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8038f20 tx=0x7fb7f0045890 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8038f20 tx=0x7fb7f0045890 comp rx=0 tx=0).stop 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c50 con 0x7fb7e804f760 2024-03-12T03:37:54.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.016+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d90 con 0x7fb8000ffcc0 2024-03-12T03:37:54.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.017+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:54.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.017+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:54.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.017+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:54.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.017+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ed0 con 0x7fb7e804f760 2024-03-12T03:37:54.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.018+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80338d0 tx=0x7fb7f8031850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:54.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f804a470 con 0x7fb7e804f760 2024-03-12T03:37:54.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804f760 2024-03-12T03:37:54.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:54.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80061d0 con 0x7fb7e804f760 2024-03-12T03:37:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:54.016+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6630 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:54.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:54.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:54.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:54.059+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:53.815+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:53.815+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:53.815+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:53.816+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:53.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:53.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:53.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:54.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:53.993+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:55.077+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:55.077+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:55.077+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:55.077+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.005+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:54.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:54.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:54.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:54.833+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:56.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:56.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:56.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:56.117+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:55.980+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:55.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:55.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:55.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:55.876+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:57.102 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:56.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:57.103 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:56.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:57.103 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:56.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:57.103 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:56.978+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:57.103 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:56.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:57.103 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:56.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:57.103 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:56.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:57.103 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:56.832+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:57.099+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:57.099+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:57.099+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:57.101+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:58.133 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:57.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:58.133 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:57.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:58.133 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:57.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:58.133 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:57.838+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:58.134 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:57.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:58.134 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:57.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:58.134 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:57.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:58.134 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:57.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:58.130+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:58.130+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:58.130+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:58.131+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:59.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:37:59.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:37:59.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80338d0 tx=0x7fb7f8031850 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80338d0 tx=0x7fb7f8031850 comp rx=0 tx=0).stop 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ba0 con 0x7fb7e804fb40 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb8000ffcc0 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.017+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.018+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.018+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:37:59.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.018+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e20 con 0x7fb7e804fb40 2024-03-12T03:37:59.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.018+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8015580 tx=0x7fb7f801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:37:59.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.018+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804fb40 2024-03-12T03:37:59.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804fb40 2024-03-12T03:37:59.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:37:59.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f802cd20 con 0x7fb7e804fb40 2024-03-12T03:37:59.114 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:37:59.017+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6689 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:37:59.114 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:58.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:59.114 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:58.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:59.114 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:58.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:37:59.114 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:58.933+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:37:59.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:58.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:59.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:58.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:59.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:58.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:37:59.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:58.833+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:37:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:59.111+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:59.111+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:59.111+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:37:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:37:59.112+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:59.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:59.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:59.813+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:37:59.814+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:59.885+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:59.885+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:59.885+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:37:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:37:59.886+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:00.145+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:00.145+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:00.145+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:00.145+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:01.119 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:00.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:01.119 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:00.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:01.119 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:00.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:01.119 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:00.932+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:01.119 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:00.816+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:01.119 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:00.816+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:01.119 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:00.816+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:01.120 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:00.817+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:01.446 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:01.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:01.447 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:01.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:01.447 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:01.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:01.447 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:01.117+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:01.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:01.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:01.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:01.825+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:01.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:01.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:01.906+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:02.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:01.907+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:02.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:02.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:02.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:02.119+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:02.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:02.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:02.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:02.802+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:02.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:02.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:02.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:03.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:02.921+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:03.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:03.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:03.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:03.076+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:03.931 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:03.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:03.932 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:03.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:03.932 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:03.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:03.932 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:03.795+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8015580 tx=0x7fb7f801c830 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8015580 tx=0x7fb7f801c830 comp rx=0 tx=0).stop 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ba0 con 0x7fb7e804f760 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb8000ffcc0 2024-03-12T03:38:04.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.018+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:04.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.019+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:04.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.019+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:04.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.019+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e20 con 0x7fb7e804f760 2024-03-12T03:38:04.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.019+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001b890 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:04.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.021+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045b60 con 0x7fb7e804f760 2024-03-12T03:38:04.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.021+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000f8c0 con 0x7fb7e804f760 2024-03-12T03:38:04.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:04.022+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071070 con 0x7fb7e804f760 2024-03-12T03:38:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:04.019+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6750 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:03.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:03.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:03.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:03.930+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:04.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:04.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:04.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:04.057+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.011+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:04.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:04.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:04.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:04.939+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:04.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:04.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:04.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:05.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:04.840+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:06.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:05.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:06.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:05.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:06.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:05.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:06.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:05.895+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:06.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:05.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:06.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:05.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:06.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:05.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:06.266 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:05.857+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:06.267 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.972+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:06.267 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.972+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:06.267 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.972+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:06.267 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:05.973+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:06.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:06.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:06.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:06.851+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:06.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:06.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:06.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:06.949+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:06.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:06.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:06.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:06.863+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:07.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:07.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:07.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:07.847+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:07.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:07.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:07.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:07.898+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:07.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:07.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:07.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:07.976+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:09.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:09.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:09.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001b890 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001b890 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b10 con 0x7fb7e804fb40 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c50 con 0x7fb8000ffcc0 2024-03-12T03:38:09.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.019+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:09.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.020+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:09.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.020+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:09.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.020+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050d90 con 0x7fb7e804fb40 2024-03-12T03:38:09.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.020+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e60 tx=0x7fb7f005ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:09.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.021+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045b60 con 0x7fb7e804fb40 2024-03-12T03:38:09.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.022+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804fb40 2024-03-12T03:38:09.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.023+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071070 con 0x7fb7e804fb40 2024-03-12T03:38:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:09.020+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6814 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:08.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:08.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:08.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:08.972+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:08.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:08.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:08.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:08.840+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:08.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:08.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:08.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:09.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:08.919+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:09.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.335+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:38:09.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:09.335+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:38:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.006+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:09.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:09.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:09.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:09.882+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:09.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:09.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:09.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:10.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:09.961+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:10.903+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:10.903+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:10.903+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:10.904+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:10.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:10.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:10.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:10.933+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:10.995+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:11.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:11.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:11.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:11.859+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:11.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:11.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:11.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:11.885+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:11.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:11.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:11.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:11.974+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:12.862+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:12.862+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:12.862+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:12.863+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:12.904+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:12.904+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:12.904+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:12.905+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:12.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:12.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:12.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:12.971+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:14.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e60 tx=0x7fb7f005ae10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009e60 tx=0x7fb7f005ae10 comp rx=0 tx=0).stop 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050bc0 con 0x7fb7e804f760 2024-03-12T03:38:14.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d00 con 0x7fb8000ffcc0 2024-03-12T03:38:14.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.021+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:14.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.022+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:14.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.022+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:14.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.022+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e40 con 0x7fb7e804f760 2024-03-12T03:38:14.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.022+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80315a0 tx=0x7fb7f002d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:14.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.023+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803b5a0 con 0x7fb7e804f760 2024-03-12T03:38:14.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.023+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804f760 2024-03-12T03:38:14.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:14.023+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f802cd20 con 0x7fb7e804f760 2024-03-12T03:38:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:14.021+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6876 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:13.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:13.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:13.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:13.818+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:13.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:13.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:13.975+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:13.976+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:13.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:13.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:13.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:14.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:13.956+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:14.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:14.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:14.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:14.923+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:14.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:14.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:14.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:14.786+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:14.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:14.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:14.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:14.995+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:15.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:15.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:15.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:15.767+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:15.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:15.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:15.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:15.968+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:16.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:15.882+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:16.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:15.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:16.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:15.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:16.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:15.883+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:16.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:16.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:16.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:16.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:16.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:16.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:16.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:16.750+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:16.910+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:16.910+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:16.910+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:16.910+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:16.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:17.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:16.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:17.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:16.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:17.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:16.937+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:17.953+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:17.953+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:17.953+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:17.954+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:17.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:17.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:17.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:17.789+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:17.924+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:17.924+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:17.924+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:17.925+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:19.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.021+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80315a0 tx=0x7fb7f002d5e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80315a0 tx=0x7fb7f002d5e0 comp rx=0 tx=0).stop 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050bc0 con 0x7fb7e804fb40 2024-03-12T03:38:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d00 con 0x7fb8000ffcc0 2024-03-12T03:38:19.025 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:18.933+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:19.026 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:18.933+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:19.026 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:18.933+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:19.026 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:18.934+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:19.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:18.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:19.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:18.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:19.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:18.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:19.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:18.992+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:19.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.022+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:19.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:18.754+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:19.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:18.754+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:19.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:18.754+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:19.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:18.754+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:19.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.023+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:19.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.023+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:19.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.023+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e40 con 0x7fb7e804fb40 2024-03-12T03:38:19.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.023+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80409d0 tx=0x7fb7f8055ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:19.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.023+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045b60 con 0x7fb7e804fb40 2024-03-12T03:38:19.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.023+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804fb40 2024-03-12T03:38:19.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:19.023+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804fb40 2024-03-12T03:38:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:19.023+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6936 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:19.995 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:19.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:19.995 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:19.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:19.995 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:19.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:19.995 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:19.932+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:19.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:19.739+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:19.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:19.739+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:19.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:19.739+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:19.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:19.740+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:19.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:19.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:19.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:19.993+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:20.977+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:20.977+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:20.977+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:20.978+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:21.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:20.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:21.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:20.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:21.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:20.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:21.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:20.746+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:21.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:20.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:21.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:20.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:21.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:20.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:21.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:20.949+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:21.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:21.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:21.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:21.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:21.953 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:21.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:21.953 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:21.785+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:21.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:21.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:21.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:21.990+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:21.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:21.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:21.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:21.950+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:23.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:23.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:23.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:23.033+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:22.971+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:22.971+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:22.971+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:22.972+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:22.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:22.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:22.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:22.826+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:24.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:24.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:24.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80409d0 tx=0x7fb7f8055ba0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:24.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80409d0 tx=0x7fb7f8055ba0 comp rx=0 tx=0).stop 2024-03-12T03:38:24.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:24.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:24.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:24.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:24.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050b10 con 0x7fb7e804f760 2024-03-12T03:38:24.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.024+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c50 con 0x7fb8000ffcc0 2024-03-12T03:38:24.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.025+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:24.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.026+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:24.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.026+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:24.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.026+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050d90 con 0x7fb7e804f760 2024-03-12T03:38:24.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.026+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802bf50 tx=0x7fb7f004cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:24.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.027+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803b5a0 con 0x7fb7e804f760 2024-03-12T03:38:24.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.027+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8031430 con 0x7fb7e804f760 2024-03-12T03:38:24.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8019bf0 con 0x7fb7e804f760 2024-03-12T03:38:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:24.025+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 6996 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:23.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:23.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:23.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:23.821+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:24.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:24.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:24.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:24.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:24.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:24.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:24.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:24.022+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:23.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:24.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:23.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:24.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:23.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:24.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:23.949+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:24.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.695+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:38:24.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:24.695+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:38:25.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:24.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:25.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:24.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:25.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:24.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:25.027 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:24.935+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:25.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:24.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:25.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:24.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:25.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:24.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:25.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:24.846+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:25.023+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:25.023+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:25.023+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:25.024+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:25.939+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:25.939+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:25.939+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:25.940+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:25.821+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:25.821+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:25.821+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:25.822+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:26.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:26.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:26.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:26.062+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:27.022 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:26.926+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:27.022 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:26.926+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:27.022 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:26.926+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:27.022 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:26.927+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:27.022 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:26.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:27.023 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:26.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:27.023 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:26.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:27.023 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:26.815+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:27.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:27.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:27.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:27.020+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:28.054 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:27.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:28.055 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:27.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:28.055 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:27.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:28.055 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:27.785+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:28.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:27.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:28.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:27.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:28.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:27.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:28.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:27.950+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:28.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:28.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:28.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:28.053+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:38:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802bf50 tx=0x7fb7f004cd10 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802bf50 tx=0x7fb7f004cd10 comp rx=0 tx=0).stop 2024-03-12T03:38:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:29.029 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:28.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:29.029 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:28.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:29.029 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:28.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:29.029 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:28.769+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:29.029 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:28.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:29.029 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:28.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:29.030 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:28.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:29.030 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:28.923+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:29.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:29.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ba0 con 0x7fb7e804fb40 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.025+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb8000ffcc0 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.026+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.026+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.026+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.026+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e20 con 0x7fb7e804fb40 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.027+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802afd0 tx=0x7fb7f8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803b5a0 con 0x7fb7e804fb40 2024-03-12T03:38:29.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804fb40 2024-03-12T03:38:29.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:29.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8019bf0 con 0x7fb7e804fb40 2024-03-12T03:38:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:29.026+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7059 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:29.092+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:29.092+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:29.092+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:29.093+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:30.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:29.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:30.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:29.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:30.111 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:29.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:30.111 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:29.816+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:30.111 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:29.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:30.111 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:29.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:30.111 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:29.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:30.111 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:29.970+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:30.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:30.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:30.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:30.108+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:30.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:30.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:30.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:31.000+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:30.785+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:30.785+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:31.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:30.785+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:31.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:30.786+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:31.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:31.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:31.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:31.076+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:31.821+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:31.821+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:31.821+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:31.821+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:32.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:31.998+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:31.998+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:31.998+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:31.998+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:32.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:32.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:32.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:32.060+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:32.849+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:32.849+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:32.849+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:32.849+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:33.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:33.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:33.037+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:33.038+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:32.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:32.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:32.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:32.988+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:33.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:33.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:33.914 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:33.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:33.914 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:33.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:33.914 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:33.804+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:38:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:34.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802afd0 tx=0x7fb7f8009b30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802afd0 tx=0x7fb7f8009b30 comp rx=0 tx=0).stop 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050c50 con 0x7fb7e804f760 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.026+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d90 con 0x7fb8000ffcc0 2024-03-12T03:38:34.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.027+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.027+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.027+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.027+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ed0 con 0x7fb7e804f760 2024-03-12T03:38:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.028+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009de0 tx=0x7fb7f8050560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:34.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803b5a0 con 0x7fb7e804f760 2024-03-12T03:38:34.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.029+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f801de40 con 0x7fb7e804f760 2024-03-12T03:38:34.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:34.029+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8019bf0 con 0x7fb7e804f760 2024-03-12T03:38:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:34.027+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7123 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:34.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:34.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:34.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:34.039+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:34.032+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:34.032+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:34.032+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:34.033+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:34.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:34.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:34.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:34.828+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:35.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:35.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:35.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:35.039+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:35.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:35.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:35.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:35.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:35.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:35.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:35.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:35.021+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:36.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:36.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:36.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:36.067+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:35.855+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:35.855+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:35.855+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:35.856+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:36.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:36.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:36.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:36.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:36.045+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:36.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:36.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:36.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:36.882+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:37.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:37.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:37.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:37.055+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:37.076+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:37.076+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:37.076+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:37.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:37.077+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:38.055+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:38.055+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:38.055+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:38.056+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.039+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:37.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:37.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:37.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:37.899+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:39.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.027+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009de0 tx=0x7fb7f8050560 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009de0 tx=0x7fb7f8050560 comp rx=0 tx=0).stop 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb7e804fb40 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e20 con 0x7fb8000ffcc0 2024-03-12T03:38:39.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.028+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:39.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.029+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:39.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.029+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:39.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.029+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f60 con 0x7fb7e804fb40 2024-03-12T03:38:39.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.029+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000dfc0 tx=0x7fb7f8055800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:39.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.030+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f0045b60 con 0x7fb7e804fb40 2024-03-12T03:38:39.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.030+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804fb40 2024-03-12T03:38:39.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:39.031+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804fb40 2024-03-12T03:38:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:39.028+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7185 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:38.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:38.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:38.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:38.869+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:39.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:39.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:39.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:39.064+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:39.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:38.992+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:40.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:40.056+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:38:40.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:40.056+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:38:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:40.047+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:40.047+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:40.047+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:40.048+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:39.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:39.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:40.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:39.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:40.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:39.855+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:40.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:40.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:40.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:40.042+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:41.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:41.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:41.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:41.054+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:40.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:40.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:40.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:40.838+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:41.010+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:41.010+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:41.010+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:41.011+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:42.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:42.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:42.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:42.062+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:41.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:41.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:41.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:41.815+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:42.042+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:42.042+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:42.042+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:42.043+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:43.058 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:42.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:43.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:42.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:43.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:42.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:43.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:42.809+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:43.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:43.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:43.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:43.056+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:43.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:43.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:43.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:43.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:43.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:43.075+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000dfc0 tx=0x7fb7f8055800 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000dfc0 tx=0x7fb7f8055800 comp rx=0 tx=0).stop 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ce0 con 0x7fb7e804f760 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e20 con 0x7fb8000ffcc0 2024-03-12T03:38:44.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.029+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.030+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.030+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:44.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.030+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f60 con 0x7fb7e804f760 2024-03-12T03:38:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.030+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0051fd0 tx=0x7fb7f0019930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.030+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804f760 2024-03-12T03:38:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.031+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804f760 2024-03-12T03:38:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:44.032+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:38:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:44.030+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7246 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:44.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:44.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:44.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:44.039+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:43.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:43.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:43.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:43.854+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:44.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:44.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:44.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:44.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:44.090+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:44.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:44.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:44.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:44.884+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:45.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:45.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:45.091+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:45.092+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:45.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:45.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:45.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:45.021+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:46.202 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:46.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:46.202 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:46.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:46.202 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:46.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:46.202 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:46.079+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:46.202 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:45.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:46.202 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:45.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:46.203 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:45.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:46.203 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:45.924+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:46.203 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:46.053+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:46.203 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:46.053+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:46.203 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:46.053+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:46.203 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:46.054+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:46.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:46.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:46.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:46.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:47.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:47.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:47.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:47.110+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:47.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:47.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:47.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:47.080+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:48.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:48.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:48.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:48.123+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:48.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:48.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:48.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:48.124+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:47.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:47.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:47.877+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:48.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:47.877+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:49.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:49.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:49.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0051fd0 tx=0x7fb7f0019930 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:49.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0051fd0 tx=0x7fb7f0019930 comp rx=0 tx=0).stop 2024-03-12T03:38:49.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d70 con 0x7fb7e804fb40 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050eb0 con 0x7fb8000ffcc0 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.030+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.031+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.031+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:49.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.031+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ff0 con 0x7fb7e804fb40 2024-03-12T03:38:49.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.031+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f0006e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:49.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.032+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804fb40 2024-03-12T03:38:49.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.032+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0053450 con 0x7fb7e804fb40 2024-03-12T03:38:49.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:49.032+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804fb40 2024-03-12T03:38:49.105 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:49.031+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7308 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:49.105 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:48.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:49.106 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:48.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:49.106 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:48.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:49.106 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:48.831+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:49.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:49.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:49.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:49.103+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:49.130+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:49.130+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:49.130+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:49.131+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:50.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:49.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:50.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:49.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:50.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:49.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:50.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:49.874+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:50.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:50.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:50.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:50.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:50.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:50.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:50.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:50.104+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:50.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:50.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:50.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:50.126+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:51.134 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:51.088+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:51.134 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:51.088+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:51.134 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:51.088+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:51.134 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:51.088+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:51.135 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:50.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:51.135 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:50.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:51.135 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:50.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:51.135 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:50.876+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:51.131+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:51.131+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:51.131+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:51.132+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:52.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:52.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:52.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:52.122+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:52.102+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:52.102+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:52.102+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:52.104+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:51.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:51.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:51.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:52.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:51.882+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:52.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:52.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:52.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:52.915+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:53.085+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:53.085+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:53.085+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:53.086+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:53.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:53.137+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:53.137+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:53.138+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:54.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f0006e40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f0006e40 comp rx=0 tx=0).stop 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050d70 con 0x7fb7e804f760 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050eb0 con 0x7fb8000ffcc0 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.031+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.032+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.032+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:54.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.032+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050ff0 con 0x7fb7e804f760 2024-03-12T03:38:54.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.032+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0010460 tx=0x7fb7f00369c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:54.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.033+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804f760 2024-03-12T03:38:54.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.033+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0053450 con 0x7fb7e804f760 2024-03-12T03:38:54.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:54.034+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:38:54.123 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:54.031+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7372 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:54.124 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:53.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:54.124 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:53.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:54.124 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:53.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:54.124 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:53.866+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:54.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:54.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:54.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:54.122+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:54.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:54.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:54.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:54.154+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:55.101 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:54.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:55.101 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:54.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:55.101 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:54.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:55.101 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:54.818+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:55.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:55.415+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:38:55.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:55.415+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:38:55.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:55.098+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:55.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:55.098+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:55.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:55.098+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:55.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:55.099+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:55.418 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:55.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:55.419 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:55.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:55.419 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:55.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:55.419 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:55.195+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:55.798+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:55.798+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:55.798+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:55.798+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:56.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:56.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:56.140+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:56.140+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:56.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:56.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:56.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:56.200+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:57.151 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:56.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:57.151 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:56.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:57.151 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:56.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:57.152 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:56.824+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:57.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:57.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:57.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:57.187+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:57.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:57.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:57.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:57.149+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:58.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:57.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:58.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:57.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:58.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:57.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:58.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:57.871+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:58.490 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:58.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:58.491 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:58.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:58.491 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:58.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:58.491 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:58.226+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:38:58.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:58.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:58.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:58.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:58.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:58.126+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:58.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:58.126+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0010460 tx=0x7fb7f00369c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0010460 tx=0x7fb7f00369c0 comp rx=0 tx=0).stop 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb7e804fb40 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050f40 con 0x7fb8000ffcc0 2024-03-12T03:38:59.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.032+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:38:59.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.033+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:38:59.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.033+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:38:59.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.033+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8051080 con 0x7fb7e804fb40 2024-03-12T03:38:59.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.033+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f4a0 tx=0x7fb7f000d8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:38:59.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.034+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f801f640 con 0x7fb7e804fb40 2024-03-12T03:38:59.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.034+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804fb40 2024-03-12T03:38:59.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:38:59.034+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8019bf0 con 0x7fb7e804fb40 2024-03-12T03:38:59.277 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:38:59.032+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7436 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:38:59.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:58.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:59.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:58.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:59.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:58.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:38:59.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:58.888+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:59.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:59.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:59.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:59.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:59.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:59.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:38:59.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:38:59.102+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:38:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:59.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:59.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:59.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:38:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:38:59.275+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:00.153 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:59.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:00.154 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:59.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:00.154 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:59.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:00.154 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:38:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:38:59.888+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:00.151+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:00.151+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:00.151+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:00.151+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:00.269+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:00.269+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:00.269+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:00.270+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:01.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:00.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:01.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:00.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:01.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:00.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:01.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:00.858+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:01.185+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:01.185+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:01.185+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:01.186+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:01.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:01.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:01.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:01.273+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:02.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:01.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:02.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:01.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:02.127 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:01.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:02.127 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:01.821+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:02.163+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:02.163+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:02.163+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:02.163+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:02.247+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:02.247+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:02.247+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:02.248+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:02.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:02.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:02.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:02.794+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:03.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:03.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:03.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:03.158+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:03.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:03.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:03.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:03.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:03.267+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:04.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:39:04.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:04.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f4a0 tx=0x7fb7f000d8a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:04.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fb7e800f4a0 tx=0x7fb7f000d8a0 comp rx=0 tx=0).stop 2024-03-12T03:39:04.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:04.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb7e804f760 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.033+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050f40 con 0x7fb8000ffcc0 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.034+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.034+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.034+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.034+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8051080 con 0x7fb7e804f760 2024-03-12T03:39:04.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.035+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017a10 tx=0x7fb7f80528c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:04.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.036+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804f760 2024-03-12T03:39:04.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.036+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0053450 con 0x7fb7e804f760 2024-03-12T03:39:04.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:04.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:39:04.145 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:04.033+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7498 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:04.145 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:03.836+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:04.145 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:03.836+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:04.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:03.836+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:04.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:03.837+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:04.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:04.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:04.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:04.249+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:04.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:04.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:04.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:04.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:04.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:04.143+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:04.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:04.143+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:05.176 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:04.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:05.177 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:04.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:05.177 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:04.884+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:05.177 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:04.885+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:05.175+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:05.175+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:05.175+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:05.175+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:05.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:05.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:05.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:05.214+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:06.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:06.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:06.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:06.148+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:06.167+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:06.167+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:06.167+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:06.167+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:05.919+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:05.919+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:05.919+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:06.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:05.920+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:07.185 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:07.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:07.185 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:07.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:07.185 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:07.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:07.185 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:07.110+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:07.185 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:07.161+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:07.185 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:07.161+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:07.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:07.161+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:07.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:07.161+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:07.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:06.917+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:07.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:06.917+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:07.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:06.917+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:07.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:06.918+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:08.126+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:08.126+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:08.126+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:08.126+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:07.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:07.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:07.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:07.942+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:08.161+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:08.161+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:08.161+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:08.162+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017a10 tx=0x7fb7f80528c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8017a10 tx=0x7fb7f80528c0 comp rx=0 tx=0).stop 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:09.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:09.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804fb40 2024-03-12T03:39:09.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:39:09.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.034+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:09.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.035+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:09.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.035+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:09.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.035+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804fb40 2024-03-12T03:39:09.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.035+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f002c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:09.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804fb40 2024-03-12T03:39:09.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b3b0 con 0x7fb7e804fb40 2024-03-12T03:39:09.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:09.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:39:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:09.034+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7562 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:09.144+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:09.144+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:09.144+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:09.144+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:09.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:09.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:09.208+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:09.209+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:08.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:08.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:08.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:08.971+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:10.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:09.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:10.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:09.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:10.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:09.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:10.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:09.954+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:10.235 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:10.152+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:10.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:10.152+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:10.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:10.152+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:10.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:10.153+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:10.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:10.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:10.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:10.233+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:10.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:10.775+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:39:10.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:10.775+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:39:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:10.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:10.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:10.992+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:10.993+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:11.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:11.582 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:11.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:11.582 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:11.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:11.582 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:11.281+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:11.582 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:11.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:11.582 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:11.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:11.582 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:11.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:11.582 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:11.115+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:12.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:12.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:12.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:12.032+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:12.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:12.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:12.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:12.265+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:12.075+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:12.075+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:12.075+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:12.076+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:13.108+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:13.108+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:13.108+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:13.109+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:13.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:13.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:13.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:13.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:13.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:13.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:13.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:13.040+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:13.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:13.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:13.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:13.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:13.229+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:14.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f002c790 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fb7f001a480 tx=0x7fb7f002c790 comp rx=0 tx=0).stop 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050920 con 0x7fb7e804f760 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb8000ffcc0 2024-03-12T03:39:14.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.035+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:14.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.036+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:14.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.036+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:14.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.036+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e00 con 0x7fb7e804f760 2024-03-12T03:39:14.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.036+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00108a0 tx=0x7fb7f00359a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:14.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f004bb40 con 0x7fb7e804f760 2024-03-12T03:39:14.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0071300 con 0x7fb7e804f760 2024-03-12T03:39:14.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:14.039+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804f760 2024-03-12T03:39:14.257 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:14.035+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7625 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:14.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:14.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:14.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:14.083+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:14.077+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:14.077+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:14.077+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:14.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:14.078+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:14.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:14.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:14.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:14.255+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:15.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:15.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:15.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:15.117+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:15.093+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:15.093+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:15.093+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:15.094+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:15.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:15.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:15.275+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:15.276+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:16.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:16.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:16.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:16.092+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:16.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:16.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:16.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:16.318+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:16.133+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:16.133+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:16.133+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:16.134+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:17.177+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:17.177+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:17.177+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:17.178+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:17.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:17.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:17.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:17.273+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:17.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:17.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:17.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:17.082+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:18.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:18.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:18.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:18.046+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:18.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:18.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:18.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:18.205+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:18.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:18.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:18.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:18.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:19.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:39:19.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:19.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00108a0 tx=0x7fb7f00359a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00108a0 tx=0x7fb7f00359a0 comp rx=0 tx=0).stop 2024-03-12T03:39:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050920 con 0x7fb7e804fb40 2024-03-12T03:39:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb8000ffcc0 2024-03-12T03:39:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.036+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.037+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.037+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.037+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e00 con 0x7fb7e804fb40 2024-03-12T03:39:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.037+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804a160 tx=0x7fb7f000b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f802f310 con 0x7fb7e804fb40 2024-03-12T03:39:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804fb40 2024-03-12T03:39:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:19.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8018120 con 0x7fb7e804fb40 2024-03-12T03:39:19.304 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:19.036+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7688 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:19.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:19.173+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:19.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:19.173+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:19.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:19.173+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:19.305 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:19.174+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:19.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:19.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:19.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:19.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:19.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:19.088+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:19.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:19.089+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:19.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:19.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:19.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:19.302+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:20.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:20.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:20.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:20.122+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:20.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:20.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:20.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:20.284+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:20.137+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:20.137+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:20.137+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:20.137+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:21.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:21.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:21.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:21.082+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:21.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:21.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:21.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:21.306+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:21.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:21.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:21.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:21.123+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:22.358 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:22.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:22.358 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:22.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:22.358 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:22.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:22.358 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:22.318+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:22.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:22.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:22.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:22.086+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:22.138+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:22.138+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:22.138+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:22.359 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:22.138+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:23.049+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:23.049+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:23.049+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:23.050+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:23.292+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:23.292+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:23.292+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:23.293+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:23.179+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:23.179+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:23.179+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:23.180+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:24.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.036+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:39:24.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:24.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804a160 tx=0x7fb7f000b1b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:24.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804a160 tx=0x7fb7f000b1b0 comp rx=0 tx=0).stop 2024-03-12T03:39:24.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:24.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:24.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:24.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:24.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804f760 2024-03-12T03:39:24.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:39:24.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.037+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:24.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.038+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:24.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.038+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:24.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.038+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804f760 2024-03-12T03:39:24.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.038+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b5d0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:24.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.039+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f001b6f0 con 0x7fb7e804f760 2024-03-12T03:39:24.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.039+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f0045820 con 0x7fb7e804f760 2024-03-12T03:39:24.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:24.041+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804f760 2024-03-12T03:39:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:24.037+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7749 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:24.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:24.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:24.243+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:24.244+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:24.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:24.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:24.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:24.190+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.033+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:25.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:25.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:25.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:25.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:25.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:25.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:25.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:25.234+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:25.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.985+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:25.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.985+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:25.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.985+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:25.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:24.986+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:25.262+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:25.262+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:25.262+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:25.262+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:26.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:26.135+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:39:26.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:26.135+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:39:26.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:25.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:26.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:25.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:26.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:25.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:26.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:25.948+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:26.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:26.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:26.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:26.233+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:26.240+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:26.240+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:26.240+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:26.241+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:27.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:27.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:27.218+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:27.219+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:26.996+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:26.996+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:26.996+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:26.997+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:27.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:27.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:27.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:27.225+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:28.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:27.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:28.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:27.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:28.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:27.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:28.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:27.985+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:28.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:28.216+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:28.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:28.216+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:28.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:28.216+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:28.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:28.217+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:28.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:28.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:28.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:28.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b5d0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fb7e804b5d0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050920 con 0x7fb7e804fb40 2024-03-12T03:39:29.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb8000ffcc0 2024-03-12T03:39:29.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.038+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:29.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.039+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:29.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.039+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:29.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.039+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e00 con 0x7fb7e804fb40 2024-03-12T03:39:29.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.039+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802afa0 tx=0x7fb7f002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:29.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803fb60 con 0x7fb7e804fb40 2024-03-12T03:39:29.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804fb40 2024-03-12T03:39:29.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:29.041+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8009750 con 0x7fb7e804fb40 2024-03-12T03:39:29.258 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:29.038+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7811 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:29.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:28.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:29.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:28.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:29.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:28.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:29.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:28.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:29.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:29.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:29.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:29.256+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:29.291+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:29.291+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:29.291+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:29.292+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:30.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:29.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:30.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:29.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:30.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:29.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:30.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:29.940+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:30.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:30.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:30.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:30.278+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:30.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:30.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:30.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:30.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:31.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:30.914+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:31.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:30.914+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:31.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:30.914+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:31.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:30.915+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:31.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:31.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:31.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:31.289+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:31.263+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:31.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:31.263+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:31.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:31.263+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:31.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:31.264+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:32.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:31.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:32.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:31.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:32.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:31.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:32.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:31.893+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:32.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:32.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:32.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:32.303+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:32.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:32.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:32.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:32.231+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:33.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:32.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:33.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:32.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:33.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:32.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:33.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:32.852+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:33.254+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:33.254+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:33.254+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:33.255+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:33.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:33.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:33.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:33.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:33.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:33.348+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:34.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:39:34.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:34.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802afa0 tx=0x7fb7f002e000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:34.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802afa0 tx=0x7fb7f002e000 comp rx=0 tx=0).stop 2024-03-12T03:39:34.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:34.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:34.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:34.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:34.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804f760 2024-03-12T03:39:34.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:39:34.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.039+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:34.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.040+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:34.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.040+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:34.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.040+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804f760 2024-03-12T03:39:34.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.040+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803ffa0 tx=0x7fb7f8049250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:34.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804f760 2024-03-12T03:39:34.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8003c90 con 0x7fb7e804f760 2024-03-12T03:39:34.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:34.041+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8009750 con 0x7fb7e804f760 2024-03-12T03:39:34.223 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:34.039+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7874 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:34.223 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:33.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:34.223 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:33.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:34.223 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:33.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:34.223 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:33.852+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:34.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:34.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:34.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:34.315+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:34.220+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:34.220+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:34.220+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:34.221+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:35.207 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:34.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:35.208 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:34.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:35.208 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:34.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:35.208 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:34.855+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:35.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:35.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:35.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:35.300+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:35.205+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:35.205+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:35.205+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:35.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:35.205+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:36.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:35.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:36.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:35.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:36.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:35.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:36.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:35.864+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:36.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:36.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:36.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:36.307+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:36.183+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:36.183+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:36.183+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:36.183+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:37.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:36.828+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:37.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:36.828+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:37.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:36.828+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:37.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:36.829+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:37.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:37.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:37.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:37.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:37.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:37.288+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:37.533 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:37.289+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:37.533 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:37.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:37.534 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:37.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:37.534 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:37.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:37.534 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:37.183+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:38.160 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:37.879+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:38.161 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:37.879+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:38.161 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:37.879+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:38.161 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:37.880+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:38.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:38.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:38.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:38.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:38.159+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:38.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:38.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:38.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:38.279+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:39.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803ffa0 tx=0x7fb7f8049250 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb7f803ffa0 tx=0x7fb7f8049250 comp rx=0 tx=0).stop 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804fb40 2024-03-12T03:39:39.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.040+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:39:39.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.041+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:39.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.041+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:39.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.041+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:39.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.041+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804fb40 2024-03-12T03:39:39.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.042+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8046de0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:39.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.043+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009a00 con 0x7fb7e804fb40 2024-03-12T03:39:39.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.043+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f80319e0 con 0x7fb7e804fb40 2024-03-12T03:39:39.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:39.043+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f8009750 con 0x7fb7e804fb40 2024-03-12T03:39:39.162 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:39.040+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7935 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:39.162 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:38.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:39.162 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:38.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:39.163 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:38.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:39.163 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:38.842+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:39.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:39.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:39.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:39.290+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:39.159+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:39.159+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:39.159+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:39.160+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:40.131 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:39.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:40.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:39.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:40.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:39.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:40.132 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:39.823+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:40.128+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:40.128+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:40.128+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:40.129+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:40.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:40.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:40.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:40.281+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:41.181 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:40.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:41.182 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:40.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:41.182 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:40.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:41.182 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:40.832+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:41.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:41.495+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:39:41.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:41.495+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:41.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:41.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:41.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:41.307+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:41.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:41.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:41.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:41.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:41.179+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:42.165 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:41.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:42.165 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:41.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:42.165 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:41.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:42.165 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:41.819+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:42.335 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:42.162+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:42.335 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:42.162+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:42.335 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:42.162+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:42.335 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:42.163+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:42.336 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:42.331+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:42.336 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:42.331+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:42.336 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:42.331+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:42.336 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:42.332+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:42.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:42.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:42.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:42.790+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:43.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:43.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:43.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:43.363+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:43.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:43.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:43.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:43.172+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:44.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8046de0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8046de0 tx=0x7fb7f8052fb0 comp rx=0 tx=0).stop 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804f760 2024-03-12T03:39:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:39:44.045 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:43.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:44.045 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:43.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:44.045 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:43.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:44.045 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:43.810+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.041+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.042+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.042+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.042+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804f760 2024-03-12T03:39:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.042+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802dfb0 tx=0x7fb7f80145c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.043+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f803eeb0 con 0x7fb7e804f760 2024-03-12T03:39:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.043+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f803eeb0 con 0x7fb7e804f760 2024-03-12T03:39:44.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:44.045+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80287b0 con 0x7fb7e804f760 2024-03-12T03:39:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:44.041+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 7997 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:44.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:44.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:44.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:44.192+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:44.797 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:44.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:44.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:44.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:44.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:44.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:44.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:44.392+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:44.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:44.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:44.795+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:44.796+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:45.147+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:45.147+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:45.147+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:45.148+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:45.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:45.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:45.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:45.380+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:45.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:45.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:45.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:45.762+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:46.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:46.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:46.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:46.391+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:46.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:46.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:46.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:46.188+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:46.773+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:46.773+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:46.773+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:46.774+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:47.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:47.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:47.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:47.419+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:47.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:47.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:47.171+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:47.171+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:47.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:47.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:47.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:47.801+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:48.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:48.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:48.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:48.383+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:48.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:48.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:48.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:48.215+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:49.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802dfb0 tx=0x7fb7f80145c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fb7f802dfb0 tx=0x7fb7f80145c0 comp rx=0 tx=0).stop 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804fb40 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:39:49.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:49.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:49.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:49.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.042+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804fb40 2024-03-12T03:39:49.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.043+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000cf40 tx=0x7fb7f800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:49.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f004bce0 con 0x7fb7e804fb40 2024-03-12T03:39:49.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f000d790 con 0x7fb7e804fb40 2024-03-12T03:39:49.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:49.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071050 con 0x7fb7e804fb40 2024-03-12T03:39:49.230 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:49.043+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8060 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:49.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:48.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:49.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:48.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:49.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:48.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:49.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:48.830+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:49.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:49.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:49.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:49.391+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:49.228+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:49.228+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:49.228+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:49.228+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:50.267 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:49.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:50.267 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:49.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:50.267 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:49.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:50.267 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:49.843+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:50.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:50.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:50.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:50.412+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:50.264+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:50.264+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:50.264+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:50.265+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:50.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:50.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:50.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:50.882+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:51.370 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:51.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:51.370 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:51.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:51.370 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:51.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:51.370 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:51.269+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:51.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:51.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:51.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:51.366+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:52.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:51.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:52.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:51.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:52.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:51.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:52.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:51.916+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:52.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:52.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:52.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:52.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:52.344+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:52.222+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:52.222+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:52.222+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:52.223+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:53.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:52.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:53.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:52.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:53.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:52.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:53.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:52.938+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:53.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:53.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:53.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:53.256+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:53.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:53.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:53.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:53.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:53.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:53.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:53.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:53.324+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000cf40 tx=0x7fb7f800c3a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000cf40 tx=0x7fb7f800c3a0 comp rx=0 tx=0).stop 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804f760 2024-03-12T03:39:54.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:39:54.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:54.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:54.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:54.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804f760 2024-03-12T03:39:54.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.044+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800b2e0 tx=0x7fb7f0010a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:54.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.045+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804f760 2024-03-12T03:39:54.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.045+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8009cc0 con 0x7fb7e804f760 2024-03-12T03:39:54.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:54.047+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f80287b0 con 0x7fb7e804f760 2024-03-12T03:39:54.256 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:54.044+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8124 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:54.256 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:53.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:54.256 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:53.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:54.256 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:53.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:54.256 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:53.899+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:54.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:54.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:54.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:54.254+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:54.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:54.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:54.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:54.344+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:55.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:54.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:55.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:54.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:55.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:54.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:55.229 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:54.929+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:55.319+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:55.319+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:55.319+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:55.319+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:55.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:55.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:55.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:55.226+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:56.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:55.942+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:56.272 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:55.942+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:56.272 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:55.942+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:56.272 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:55.942+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:56.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:56.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:56.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:56.273+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:56.269+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:56.269+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:56.269+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:56.270+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:56.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:56.855+0000 7fb7feffd700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 msgr2=0x7fb7e803b6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.177:6828/588231664 2024-03-12T03:39:56.857 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:56.855+0000 7fb7feffd700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:6828/588231664,v1:172.21.15.177:6829/588231664] conn(0x7fb7e8039230 0x7fb7e803b6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-12T03:39:57.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:56.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:57.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:56.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:57.277 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:56.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:57.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:56.954+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:57.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:57.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:57.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:57.275+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:57.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:57.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:57.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:57.305+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:57.997+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:57.997+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:57.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:57.998+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:58.452 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:58.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:58.452 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:58.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:58.452 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:58.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:58.452 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:58.279+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:58.452 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:58.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:58.452 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:58.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:58.453 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:58.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:58.453 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:58.279+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:39:59.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-12T03:39:59.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fffff700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:39:59.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800b2e0 tx=0x7fb7f0010a80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:39:59.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fffff700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb7f800b2e0 tx=0x7fb7f0010a80 comp rx=0 tx=0).stop 2024-03-12T03:39:59.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 msgr2=0x7fb7e80503e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:39:59.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:59.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e804a270 con 0x7fb7e804fb40 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.044+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb8000ffcc0 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.045+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.045+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.045+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.045+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050e00 con 0x7fb7e804fb40 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.045+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0020000 tx=0x7fb7f8004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.046+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f00712f0 con 0x7fb7e804fb40 2024-03-12T03:39:59.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.046+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b030 con 0x7fb7e804fb40 2024-03-12T03:39:59.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:39:59.048+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804fb40 2024-03-12T03:39:59.149 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:39:59.045+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8186 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:39:59.149 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:58.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:59.149 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:58.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:59.149 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:58.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:39:59.150 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:39:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:39:58.972+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:59.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:59.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:59.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:39:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:39:59.300+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:39:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:59.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:59.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:59.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:39:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:39:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:39:59.243+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:00.199 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:00.003+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:00.200 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:00.003+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:00.200 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:00.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:00.200 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:00.004+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:00.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:00.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:00.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:00.313+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:00.197+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:00.197+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:00.197+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:00.198+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:01.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:01.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:01.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:01.176+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:01.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:01.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:01.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:01.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:01.019+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:01.342+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:01.342+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:01.342+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:01.343+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:02.374 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:02.216+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:02.374 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:02.216+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:02.374 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:02.216+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:02.374 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:02.216+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:02.374 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:02.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:02.374 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:02.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:02.374 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:02.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:02.375 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:02.067+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:02.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:02.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:02.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:02.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:02.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:02.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:02.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:02.372+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:03.341 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:03.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:03.341 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:03.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:03.341 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:03.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:03.341 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:03.238+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:03.342 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:03.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:03.342 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:03.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:03.342 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:03.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:03.342 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:03.083+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:03.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:03.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:03.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:03.339+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:04.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-12T03:40:04.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-12T03:40:04.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0020000 tx=0x7fb7f8004000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-12T03:40:04.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0020000 tx=0x7fb7f8004000 comp rx=0 tx=0).stop 2024-03-12T03:40:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 msgr2=0x7fb7e804f220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-12T03:40:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804fb40 0x7fb7e804f220 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:40:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:40:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7fcff9700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-12T03:40:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050ac0 con 0x7fb7e804f760 2024-03-12T03:40:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.045+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e8050e00 con 0x7fb8000ffcc0 2024-03-12T03:40:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.046+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-12T03:40:04.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.046+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 msgr2=0x7fb7e804a890 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-12T03:40:04.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.046+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] conn(0x7fb8000ffcc0 0x7fb7e804a890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-12T03:40:04.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.046+0000 7fb7ff7fe700 1 -- 172.21.15.118:0/2308702459 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=3+}) v3 -- 0x7fb7e8050f40 con 0x7fb7e804f760 2024-03-12T03:40:04.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.047+0000 7fb7ff7fe700 1 --2- 172.21.15.118:0/2308702459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7e804f760 0x7fb7e80503e0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0048c30 tx=0x7fb7f000bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-12T03:40:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.048+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb7f00712f0 con 0x7fb7e804f760 2024-03-12T03:40:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.048+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f001b3a0 con 0x7fb7e804f760 2024-03-12T03:40:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:40:04.048+0000 7fb7fcff9700 1 -- 172.21.15.118:0/2308702459 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mgrmap(e 23) v1 ==== 46247+0+0 (secure 0 0 0) 0x7fb7f0071040 con 0x7fb7e804f760 2024-03-12T03:40:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:04.046+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8249 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:04.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:04.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:04.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:04.259+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:04.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:04.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:04.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:04.066+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:04.470 INFO:teuthology.orchestra.run.smithi118.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-03-12T03:40:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:04.316+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:04.316+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:04.316+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:04.317+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:05.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:05.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:05.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:05.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:05.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:05.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:05.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:05.065+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:05.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-12T03:40:05.483 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:40:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:05.355+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:05.355+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:05.355+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:05.357+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:05.309+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:05.309+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:05.309+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:05.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:05.309+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:06.282 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:06.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:06.282 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:06.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:06.282 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:06.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:06.282 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:06.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:06.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:06.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:06.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:06.400+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:06.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:06.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:06.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:06.280+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:07.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:07.127+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:07.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:07.127+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:07.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:07.127+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:07.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:07.128+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:07.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:07.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:07.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:07.300+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:07.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:07.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:07.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:07.413+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:08.254 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:08.087+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:08.255 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:08.087+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:08.255 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:08.087+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:08.255 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:08.088+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:08.251+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:08.251+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:08.251+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:08.253+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:08.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:08.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:08.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:08.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:08.410+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:09.238 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:09.047+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8312 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:09.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:09.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:09.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:09.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:09.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:09.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:09.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:09.045+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:09.545 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:09.235+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:09.545 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:09.235+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:09.545 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:09.235+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:09.545 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:09.236+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:09.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:09.423+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:09.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:09.423+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:09.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:09.423+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:09.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:09.425+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:10.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:10.019+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:10.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:10.019+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:10.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:10.019+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:10.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:10.020+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:10.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:10.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:10.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:10.428+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:10.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:10.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:10.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:10.250+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.009+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.010+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:11.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:11.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:11.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:11.281+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:11.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:11.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:11.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:11.417+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:12.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:12.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:12.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:12.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:11.983+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:12.432+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:12.432+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:12.432+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:12.432+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:12.271+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:12.271+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:12.271+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:12.271+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:13.238 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:13.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:13.238 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:13.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:13.238 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:13.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:13.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:13.031+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:13.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:13.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:13.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:13.460+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:13.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:13.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:13.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:13.236+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:14.310 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:14.048+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8375 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:14.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:14.283+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:14.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:14.283+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:14.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:14.283+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:14.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:14.284+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:14.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:14.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:14.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:14.040+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:14.457+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:14.457+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:14.457+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:14.458+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:15.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:15.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:15.358 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:15.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:15.358 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:15.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:15.358 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:15.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:15.358 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:15.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:15.358 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:15.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:15.358 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:15.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:15.358 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:15.243+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:15.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:15.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:15.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:15.448+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:16.461 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:16.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:16.461 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:16.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:16.461 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:16.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:16.461 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:16.100+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:16.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:16.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:16.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:16.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:16.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:16.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:16.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:16.268+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:16.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:16.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:16.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:16.459+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:17.496 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:17.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:17.496 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:17.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:17.496 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:17.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:17.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:17.105+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:17.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:17.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:17.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:17.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:17.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:17.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:17.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:17.275+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:17.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:17.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:17.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:17.495+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:18.261 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:18.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:18.261 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:18.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:18.261 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:18.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:18.261 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:18.155+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:18.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:18.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:18.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:18.468+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:18.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:18.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:18.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:18.259+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:19.273 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:19.049+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8436 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:19.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:19.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:19.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:19.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:19.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:19.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:19.274 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:19.119+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:19.434+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:19.434+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:19.434+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:19.435+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:19.271+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:19.271+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:19.271+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:19.272+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:20.453 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:20.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:20.453 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:20.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:20.453 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:20.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:20.454 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:20.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:20.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:20.235+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:20.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:20.235+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:20.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:20.235+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:20.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:20.235+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:20.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:20.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:20.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:20.451+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:21.461 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:21.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:21.461 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:21.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:21.461 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:21.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:21.462 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:21.152+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:21.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:21.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:21.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:21.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:21.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:21.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:21.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:21.234+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:21.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:21.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:21.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:21.459+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:22.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:22.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:22.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:22.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:22.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:22.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:22.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:22.231+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:22.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:22.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:22.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:22.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:22.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:22.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:22.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:22.190+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:22.495+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:22.495+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:22.495+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:22.496+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:23.194+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:23.194+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:23.194+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:23.194+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:23.202+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:23.202+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:23.202+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:23.203+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:23.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:23.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:23.450+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:23.451+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:24.050+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8498 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:24.190+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:24.190+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:24.190+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:24.190+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:24.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:24.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:24.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:24.180+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:24.756 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:24.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:24.756 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:24.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:24.756 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:24.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:24.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:24.472+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:25.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:25.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:25.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:25.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:25.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:25.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:25.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:25.157+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:25.426 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:25.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:25.426 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:25.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:25.426 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:25.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:25.426 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:25.156+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:25.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:25.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:25.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:25.522+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:26.551 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:26.143+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:26.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:26.143+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:26.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:26.143+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:26.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:26.144+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:26.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:26.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:26.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:26.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:26.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:26.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:26.552 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:26.184+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:26.549+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:26.549+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:26.549+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:26.549+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:27.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:27.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:27.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:27.224+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:27.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:27.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:27.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:27.503 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:27.194+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:27.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:27.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:27.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:27.501+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:28.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:28.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:28.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:28.200+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:28.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:28.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:28.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:28.198+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:28.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:28.199+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:28.498+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:28.498+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:28.498+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:28.499+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:29.239 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:29.050+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8560 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:29.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:29.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:29.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:29.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:29.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:29.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:29.239 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:29.176+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:29.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:29.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:29.467+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:29.468+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:29.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:29.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:29.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:29.237+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:30.191+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:30.191+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:30.191+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:30.192+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:30.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:30.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:30.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:30.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:30.268+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:30.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:30.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:30.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:30.494+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:31.544 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:31.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:31.544 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:31.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:31.544 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:31.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:31.544 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:31.281+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:31.544 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:31.237+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:31.544 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:31.237+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:31.544 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:31.237+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:31.545 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:31.238+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:31.541+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:31.541+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:31.541+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:31.542+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:32.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:32.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:32.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:32.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:32.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:32.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:32.547 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:32.211+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:32.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:32.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:32.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:32.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:32.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:32.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:32.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:32.262+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:32.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:32.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:32.544+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:32.545+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:33.205+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:33.205+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:33.205+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:33.206+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:33.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:33.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:33.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:33.238+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:33.557+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:33.557+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:33.557+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:33.558+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:34.052+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8624 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:34.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:34.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:34.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:34.280+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:34.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:34.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:34.215+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:34.215+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:34.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:34.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:34.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:34.540+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:35.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:35.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:35.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:35.324+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:35.212+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:35.212+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:35.212+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:35.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:35.213+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:35.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:35.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:35.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:35.575+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:36.324+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:36.324+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:36.324+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:36.325+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:36.164+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:36.164+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:36.164+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:36.164+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:36.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:36.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:36.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:36.586+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:37.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:37.302+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:37.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:37.302+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:37.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:37.302+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:37.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:37.304+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:37.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:37.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:37.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:37.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:37.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:37.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:37.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:37.177+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:37.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:37.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:37.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:37.539+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:38.518 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:38.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:38.518 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:38.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:38.518 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:38.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:38.519 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:38.177+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:38.519 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:38.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:38.519 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:38.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:38.519 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:38.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:38.519 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:38.317+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:38.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:38.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:38.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:38.516+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:39.285 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:39.052+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8687 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:39.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:39.149+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:39.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:39.149+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:39.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:39.149+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:39.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:39.149+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:39.507+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:39.507+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:39.507+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:39.508+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:39.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:39.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:39.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:39.283+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:40.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:40.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:40.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:40.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:40.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:40.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:40.497 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:40.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:40.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:40.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:40.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:40.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:40.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:40.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:40.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:40.301+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:40.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:40.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:40.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:40.495+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:41.523 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:41.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:41.524 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:41.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:41.524 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:41.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:41.524 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:41.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:41.524 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:41.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:41.524 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:41.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:41.524 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:41.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:41.524 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:41.317+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:41.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:41.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:41.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:41.522+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:42.512 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:42.276+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:42.513 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:42.276+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:42.513 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:42.276+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:42.513 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:42.277+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:42.513 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:42.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:42.513 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:42.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:42.513 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:42.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:42.513 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:42.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:42.509+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:42.509+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:42.509+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:42.510+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:43.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:43.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:43.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:43.289+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:43.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:43.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:43.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:43.137+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:43.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:43.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:43.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:43.557+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:44.052+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8750 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:44.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:44.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:44.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:44.092+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:44.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:44.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:44.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:44.253+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:44.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:44.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:44.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:44.523+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:45.043+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:45.043+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:45.043+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:45.044+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:45.211+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:45.211+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:45.211+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:45.212+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:45.492+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:45.492+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:45.492+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:45.492+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:46.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:46.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:46.200+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:46.201+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:46.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:46.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:46.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:46.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:46.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:46.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:46.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:46.016+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:46.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:46.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:46.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:46.470+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.014+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:47.165+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:47.165+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:47.165+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:47.166+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:47.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:47.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:47.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:47.449+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:47.980+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:48.129+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:48.129+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:48.129+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:48.130+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:48.445+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:48.445+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:48.445+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:48.446+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:49.054+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8814 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:49.132+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:49.132+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:49.132+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:49.132+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:48.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:48.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:48.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:48.987+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:49.477+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:49.477+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:49.477+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:49.478+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:49.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:49.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:49.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:49.938+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:50.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:50.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:50.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:50.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:50.141+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:50.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:50.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:50.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:50.516+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:50.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:50.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:50.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:50.952+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:51.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:51.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:51.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:51.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:51.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:51.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:51.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:51.104+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:51.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:51.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:51.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:51.492+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:51.904+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:51.905+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:51.905+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:51.905+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:52.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:52.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:52.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:52.070+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:52.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:52.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:52.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:52.535+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:53.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:52.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:53.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:52.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:53.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:52.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:53.110 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:52.857+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:53.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:53.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:53.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:53.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:53.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:53.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:53.541 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:53.108+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:53.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:53.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:53.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:53.539+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:54.161 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:54.055+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8879 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:54.161 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:53.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:54.161 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:53.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:54.161 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:53.874+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:54.161 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:53.875+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:54.521 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:54.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:54.521 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:54.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:54.521 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:54.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:54.521 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:54.159+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:54.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:54.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:54.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:54.519+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:55.210 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:54.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:55.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:54.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:55.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:54.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:55.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:54.826+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:55.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:55.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:55.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:55.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:55.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:55.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:55.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:55.208+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:55.549+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:55.549+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:55.549+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:55.550+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:55.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:55.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:55.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:55.790+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:56.507 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:56.207+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:56.507 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:56.207+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:56.507 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:56.207+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:56.508 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:56.208+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:56.504+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:56.504+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:56.504+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:56.505+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:57.201 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:56.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:57.202 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:56.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:57.202 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:56.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:57.202 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:56.827+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:57.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:57.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:57.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:57.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:57.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:57.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:57.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:57.200+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:57.786 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:57.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:57.786 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:57.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:57.786 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:57.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:57.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:57.477+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:57.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:57.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:57.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:57.858+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:58.508 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:58.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:58.508 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:58.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:58.508 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:58.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:58.508 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:58.249+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:40:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:58.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:58.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:58.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:58.506+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:59.232 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:40:59.056+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 8942 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:40:59.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:58.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:59.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:58.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:59.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:58.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:40:59.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:58.828+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:59.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:59.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:59.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:40:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:40:59.471+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:40:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:59.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:59.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:59.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:40:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:40:59.231+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:59.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:59.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:59.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:40:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:40:59.793+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:00.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:00.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:00.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:00.449+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:00.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:00.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:00.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:00.230+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:00.804+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:00.804+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:00.804+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:00.805+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:01.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:01.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:01.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:01.274+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:01.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:01.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:01.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:01.471+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:01.882 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:01.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:01.882 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:01.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:01.883 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:01.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:01.883 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:01.803+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:02.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:02.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:02.289+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:02.289+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:02.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:02.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:02.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:02.452+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:02.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:02.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:02.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:02.787+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:03.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:03.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:03.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:03.463+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:03.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:03.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:03.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:03.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:03.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:04.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:03.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:04.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:03.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:04.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:03.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:04.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:03.813+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:04.057+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9006 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:04.264+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:04.264+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:04.264+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:04.265+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:04.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:04.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:04.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:04.439+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:05.298 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:04.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:05.298 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:04.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:05.298 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:04.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:05.298 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:04.854+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:05.295+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:05.295+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:05.295+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:05.296+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:05.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:05.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:05.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:05.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:05.400+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:06.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:05.872+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:06.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:05.872+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:06.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:05.872+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:06.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:05.873+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:06.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:06.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:06.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:06.289+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:06.447+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:06.447+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:06.447+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:06.448+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:06.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:06.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:06.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:06.876+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:07.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:07.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:07.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:07.408+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:07.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:07.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:07.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:07.319+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:08.294 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:07.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:08.294 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:07.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:08.294 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:07.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:08.294 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:07.834+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:08.291+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:08.291+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:08.291+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:08.293+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:08.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:08.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:08.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:08.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:08.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:08.390+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:09.058+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9070 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:09.329 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:08.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:09.329 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:08.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:09.329 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:08.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:09.329 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:08.815+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:09.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:09.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:09.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:09.336+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:09.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:09.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:09.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:09.419+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:10.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:09.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:10.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:09.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:10.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:09.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:10.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:09.841+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:10.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:10.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:10.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:10.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:10.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:10.444+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:10.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:10.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:10.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:10.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:10.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:10.302+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:10.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:10.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:10.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:11.283 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:10.881+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:11.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:11.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:11.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:11.472+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:11.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:11.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:11.280+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:11.281+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:12.248 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:11.912+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:12.248 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:11.912+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:12.248 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:11.912+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:12.248 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:11.912+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:12.515 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:12.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:12.515 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:12.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:12.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:12.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:12.516 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:12.246+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:12.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:12.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:12.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:12.514+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:13.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:12.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:13.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:12.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:13.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:12.931+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:13.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:12.932+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:13.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:13.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:13.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:13.516+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:13.281+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:13.281+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:13.281+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:13.282+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:14.257 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:14.059+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9134 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:13.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:13.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:13.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:13.919+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:14.557 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:14.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:14.557 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:14.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:14.557 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:14.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:14.557 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:14.256+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:14.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:14.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:14.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:14.555+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:15.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:14.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:15.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:14.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:15.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:14.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:15.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:14.884+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:15.293+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:15.293+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:15.293+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:15.294+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:15.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:15.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:15.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:15.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:15.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:15.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:15.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:15.594+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:16.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:15.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:16.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:15.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:16.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:15.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:16.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:15.921+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:16.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:16.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:16.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:16.269+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:16.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:16.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:16.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:16.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:16.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:16.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:16.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:16.625+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:17.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:16.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:17.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:16.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:17.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:16.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:17.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:16.887+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:17.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:17.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:17.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:17.256+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:17.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:17.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:17.885 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:17.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:17.885 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:17.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:17.885 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:17.597+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:18.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:17.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:18.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:17.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:18.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:17.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:18.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:17.882+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:18.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:18.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:18.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:18.215+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:18.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:18.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:18.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:18.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:18.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:18.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:18.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:18.560+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:19.174 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:19.060+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9197 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:19.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:18.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:19.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:18.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:19.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:18.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:19.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:18.887+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:19.172+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:19.172+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:19.172+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:19.173+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:20.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:19.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:20.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:19.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:20.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:19.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:20.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:19.594+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:20.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:19.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:20.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:19.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:20.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:19.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:20.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:19.838+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:20.148+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:20.148+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:20.148+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:20.149+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:20.872 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:20.613+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:20.872 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:20.613+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:20.872 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:20.613+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:20.872 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:20.614+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:21.178 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:20.869+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:21.178 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:20.869+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:21.178 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:20.869+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:21.178 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:20.870+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:21.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:21.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:21.175+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:21.176+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:21.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:21.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:21.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:21.893+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:21.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:21.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:21.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:21.650+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:22.219+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:22.219+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:22.219+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:22.220+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:22.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:22.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:22.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:22.933+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:22.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:23.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:22.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:23.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:22.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:23.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:22.695+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:23.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:23.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:23.230+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:23.230+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:23.959 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:23.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:23.959 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:23.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:23.959 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:23.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:23.959 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:23.651+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:24.211 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:24.061+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9261 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:24.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:23.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:24.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:23.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:24.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:23.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:24.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:23.957+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:24.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:24.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:24.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:24.209+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:24.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:24.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:24.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:24.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:24.958 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:24.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:24.958 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:24.670+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:24.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:24.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:24.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:24.955+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:25.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:25.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:25.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:25.181+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:25.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:25.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:25.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:25.713+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:25.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:26.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:25.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:26.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:25.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:26.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:25.952+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:26.183+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:26.183+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:26.183+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:26.184+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:26.914+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:26.914+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:26.914+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:26.915+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:26.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:26.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:26.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:26.667+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:27.163+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:27.163+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:27.163+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:27.164+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:27.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:27.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:27.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:27.681+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:27.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:28.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:27.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:28.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:27.915+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:28.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:27.916+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:28.473 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:28.135+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:28.473 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:28.135+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:28.473 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:28.135+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:28.473 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:28.137+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:28.660+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:28.660+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:28.660+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:28.660+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:29.185 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:29.062+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9324 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:29.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:28.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:29.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:28.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:29.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:28.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:29.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:28.912+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:29.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:29.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:29.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:29.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:29.183+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:29.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:29.622+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:29.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:29.622+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:29.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:29.622+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:29.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:29.623+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:29.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:29.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:29.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:29.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:30.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:30.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:30.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:30.155+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:30.901 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:30.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:30.901 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:30.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:30.901 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:30.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:30.901 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:30.586+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:30.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:30.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:30.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:30.899+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:31.146+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:31.146+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:31.146+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:31.147+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:31.933 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:31.560+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:31.933 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:31.560+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:31.933 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:31.560+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:31.933 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:31.561+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:32.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:32.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:32.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:32.140+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:31.930+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:31.930+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:31.930+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:32.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:31.931+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:32.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:32.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:32.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:32.535+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:33.098+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:33.098+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:33.098+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:33.098+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:32.957+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:32.957+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:32.957+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:33.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:32.957+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:33.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:33.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:33.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:33.514+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:34.066 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:33.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:34.066 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:33.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:34.066 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:33.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:34.066 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:33.939+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:34.549 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:34.063+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9388 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:34.549 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:34.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:34.549 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:34.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:34.550 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:34.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:34.550 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:34.115+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:34.546+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:34.546+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:34.546+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:34.548+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:35.159+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:35.159+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:35.159+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:35.160+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:34.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:34.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:34.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:34.949+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:35.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:35.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:35.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:35.501+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:36.211 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:36.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:36.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:35.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:36.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:35.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:36.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:35.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:36.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:35.959+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:36.474 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:36.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:36.475 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:36.208+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:36.475 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:36.209+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:36.472+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:36.472+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:36.472+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:36.473+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:37.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:36.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:37.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:36.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:37.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:36.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:37.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:36.941+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:37.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:37.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:37.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:37.454+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:37.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:37.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:37.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:37.230+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:38.221 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:37.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:38.221 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:37.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:38.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:37.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:38.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:37.952+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:38.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:38.404+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:38.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:38.404+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:38.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:38.404+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:38.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:38.405+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:38.550 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:38.218+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:38.550 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:38.218+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:38.550 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:38.218+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:38.551 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:38.219+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:39.064+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9450 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:39.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:39.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:39.199+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:39.200+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:38.974+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:38.974+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:38.974+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:38.975+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:39.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:39.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:39.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:39.377+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:40.174+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:40.174+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:40.174+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:40.175+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:39.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:40.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:39.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:40.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:39.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:40.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:39.999+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:40.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:40.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:40.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:40.377+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:41.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:41.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:41.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:41.182+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:41.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:41.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:41.045+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:41.045+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:41.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:41.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:41.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:41.333+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:42.195+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:42.195+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:42.195+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:42.196+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:42.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.013+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:42.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:42.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:42.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:42.334+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:43.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:43.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:43.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:43.242 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:42.972+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:43.286+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:43.286+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:43.286+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:43.287+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:43.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:43.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:43.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:43.240+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:44.303 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:44.065+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9513 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:44.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:44.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:44.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:44.266+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:44.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:44.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:44.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:44.304 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:44.018+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:44.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:44.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:44.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:44.302+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:45.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:45.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:45.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:45.237+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:45.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:45.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:45.258+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:45.258+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:45.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.021+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:45.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.021+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:45.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.021+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:45.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.021+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.989+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.989+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:46.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.989+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:46.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:45.990+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:46.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:46.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:46.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:46.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:46.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:46.211+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:46.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:46.212+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:46.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:46.206+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:46.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:46.206+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:46.314 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:46.206+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:46.314 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:46.207+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:47.220 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:47.163+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:47.220 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:47.163+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:47.220 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:47.163+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:47.220 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:47.164+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:47.220 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:47.038+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:47.220 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:47.038+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:47.221 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:47.038+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:47.221 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:47.039+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:47.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:47.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:47.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:47.218+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.042+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:48.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:48.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:48.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:48.137+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:48.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:48.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:48.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:48.191+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:49.066+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9576 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:49.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:49.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:49.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:49.179+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:48.998+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:49.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:49.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:49.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:49.219+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:50.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:50.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:50.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:50.002+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:50.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:50.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:50.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:50.238+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:50.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:50.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:50.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:50.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:50.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:50.140+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:50.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:50.141+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:51.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:51.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:51.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:51.271+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:51.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:51.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:51.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:51.029+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:51.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:51.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:51.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:51.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:51.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:51.187+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:51.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:51.188+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:52.284 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:52.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:52.284 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:52.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:52.285 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:52.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:52.285 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:52.225+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:52.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:52.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:52.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:52.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:52.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:52.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:52.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:52.014+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:52.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:52.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:52.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:52.282+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:53.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:53.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:53.260+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:53.261+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:53.062+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:53.062+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:53.062+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:53.063+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:53.221+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:53.221+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:53.221+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:53.221+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:54.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:54.067+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9640 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:54.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:54.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:54.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:54.226+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:54.261+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:54.261+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:54.261+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:54.261+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:54.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:54.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:54.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:54.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:54.102+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:55.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:55.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:55.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:55.232+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:55.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:55.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:55.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:55.300+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:55.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:55.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:55.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:55.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:56.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:56.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:56.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:56.067+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:56.222+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:56.222+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:56.222+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:56.222+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:56.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:56.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:56.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:56.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:56.293+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:57.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:57.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:57.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:57.306+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:57.079+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:57.079+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:57.079+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:57.080+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:57.217+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:57.217+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:57.217+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:57.218+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:58.085+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:58.085+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:58.085+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:58.085+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:58.261+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:58.261+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:58.261+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:58.263+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:58.262+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:58.262+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:58.262+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:58.263+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:41:59.230 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:41:59.068+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9697 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:41:59.230 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:59.113+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:59.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:59.113+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:59.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:59.113+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:41:59.231 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:41:59.114+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:59.227+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:59.227+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:59.227+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:41:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:41:59.228+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:41:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:59.300+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:59.300+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:59.300+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:41:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:41:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:41:59.301+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:00.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:00.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:00.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:00.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:00.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:00.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:00.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:00.319+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:00.253+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:00.253+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:00.253+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:00.254+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:01.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:01.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:01.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:01.042+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:01.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:01.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:01.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:01.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:01.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:01.231+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:01.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:01.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:01.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:01.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:01.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:01.352+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:02.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:02.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:02.191+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:02.192+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:02.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:02.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:02.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:02.028+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:02.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:02.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:02.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:02.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:02.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:02.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:02.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:02.394+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:03.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:03.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:03.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:03.013+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:03.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:03.142+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:03.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:03.142+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:03.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:03.142+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:03.425 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:03.143+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:03.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:03.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:03.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:03.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:03.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:03.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:03.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:03.423+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:04.069+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9748 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:04.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:04.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:04.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:04.137+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:04.042+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:04.042+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:04.042+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:04.042+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:04.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:04.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:04.404+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:04.405+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:05.396 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:05.130+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:05.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:05.130+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:05.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:05.130+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:05.397 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:05.131+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:05.397 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:05.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:05.397 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:05.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:05.397 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:05.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:05.397 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:05.093+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:05.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:05.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:05.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:05.394+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:06.137 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:06.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:06.137 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:06.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:06.137 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:06.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:06.137 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:06.047+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:06.395 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:06.134+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:06.395 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:06.134+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:06.395 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:06.134+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:06.395 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:06.135+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:06.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:06.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:06.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:06.393+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:07.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:07.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:07.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:07.032+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:07.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:07.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:07.138+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:07.139+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:07.353+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:07.353+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:07.353+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:07.354+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:08.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:08.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:08.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:08.047+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:08.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:08.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:08.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:08.169+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:08.774 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:08.331+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:08.774 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:08.331+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:08.775 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:08.331+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:08.775 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:08.332+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:09.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:09.070+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9799 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:09.087+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:09.087+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:09.087+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:09.088+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:09.299+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:09.299+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:09.299+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:09.300+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:09.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:09.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:09.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:09.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:09.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:09.182+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:10.340 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:10.174+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:10.340 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:10.174+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:10.340 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:10.174+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:10.341 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:10.174+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:10.341 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:10.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:10.341 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:10.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:10.341 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:10.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:10.341 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:10.068+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:10.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:10.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:10.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:10.338+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.028+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:11.199+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:11.199+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:11.199+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:11.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:11.200+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:11.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:11.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:11.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:11.310+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:11.989+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:12.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:12.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:12.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:12.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:12.193+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:12.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:12.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:12.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:12.313+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.027+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:13.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:13.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:13.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:13.193+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:13.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:13.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:13.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:13.266+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:14.071+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9850 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:14.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:14.221+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:14.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:14.221+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:14.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:14.221+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:14.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:14.222+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:14.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:14.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:14.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:14.222+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.997+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.997+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.997+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:13.998+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:15.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:15.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:15.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:15.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:15.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:15.182+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:15.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:15.183+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:15.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:15.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:15.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:15.047+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:15.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:15.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:15.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:15.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:15.260+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:16.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:16.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:16.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:16.073+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:16.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:16.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:16.301+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:16.302+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:16.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:16.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:16.189+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:16.190+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:17.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:17.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:17.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:17.037+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:17.197+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:17.197+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:17.197+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:17.198+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:17.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:17.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:17.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:17.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:18.356 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:18.152+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:18.357 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:18.152+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:18.357 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:18.152+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:18.357 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:18.152+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:18.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:18.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:18.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:18.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:18.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:18.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:18.357 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:18.070+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:18.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:18.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:18.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:18.355+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:19.180 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:19.072+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9900 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:19.180 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:19.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:19.180 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:19.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:19.181 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:19.082+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:19.181 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:19.083+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:19.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:19.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:19.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:19.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:19.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:19.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:19.178+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:19.178+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:20.217+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:20.217+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:20.217+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:20.217+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:20.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:20.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:20.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:20.346 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:20.070+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:20.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:20.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:20.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:20.344+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:21.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:21.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:21.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:21.025+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:21.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:21.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:21.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:21.169+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:21.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:21.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:21.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:21.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:21.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:21.352+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:22.179+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:22.179+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:22.179+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:22.180+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:22.050+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:22.050+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:22.050+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:22.050+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:22.348+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:22.348+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:22.348+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:22.348+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:23.166+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:23.166+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:23.166+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:23.166+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:23.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:23.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:23.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:23.040+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:23.579 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:23.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:23.580 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:23.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:23.580 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:23.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:23.580 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:23.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:24.148 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:24.073+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 9951 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:24.149 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:24.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:24.149 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:24.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:24.149 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:24.068+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:24.149 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:24.069+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:24.146+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:24.146+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:24.146+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:24.146+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:24.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:24.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:24.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:24.367+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:25.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:25.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:25.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:25.117+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:25.337+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:25.337+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:25.337+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:25.338+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:25.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:25.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:25.103+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:25.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:25.103+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:26.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:26.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:26.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:26.117+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:26.060+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:26.060+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:26.060+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:26.364 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:26.060+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:26.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:26.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:26.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:26.362+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.032+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:27.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:27.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:27.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:27.163+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:27.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:27.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:27.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:27.359+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:28.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:28.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:28.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:28.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:28.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:28.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:28.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:27.984+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:28.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:28.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:28.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:28.381+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:29.074+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10002 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:29.147+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:29.147+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:29.147+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:29.148+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:29.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:29.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:29.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:29.008+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:29.368+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:29.368+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:29.368+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:29.369+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:30.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:30.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:30.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:30.131+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.004+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:30.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:30.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:30.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:30.341+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:30.989+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:31.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:31.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:31.095+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:31.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:31.096+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:31.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:31.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:31.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:31.381+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:32.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:32.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:32.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:32.134+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:31.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:31.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:31.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:31.958+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:32.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:32.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:32.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:32.354+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:32.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:32.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:32.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:32.997+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:33.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:33.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:33.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:33.182+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:33.580 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:33.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:33.580 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:33.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:33.580 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:33.317+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:33.581 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:33.318+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:34.075+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10052 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:34.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:34.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:34.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:34.194+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:34.044+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:34.044+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:34.044+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:34.045+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:34.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:34.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:34.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:34.329+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:35.296 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:35.018+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:35.296 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:35.018+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:35.296 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:35.018+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:35.296 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:35.019+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:35.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:35.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:35.297 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:35.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:35.297 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:35.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:35.297 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:35.175+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:35.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:35.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:35.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:35.294+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:36.273+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:36.273+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:36.273+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:36.274+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:36.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:36.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:36.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:36.214+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:36.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.040+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:36.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.040+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:36.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.040+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:36.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.041+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:37.255 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:37.255 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:37.255 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:37.255 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:36.992+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:37.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:37.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:37.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:37.253+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:37.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:37.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:37.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:37.264+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:38.296 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:37.966+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:38.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:37.966+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:38.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:37.966+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:38.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:37.967+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:38.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:38.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:38.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:38.294+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:38.297+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:38.297+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:38.297+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:38.298+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:39.076+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10104 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:38.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:38.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:38.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:38.926+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:39.673 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:39.324+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:39.674 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:39.324+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:39.674 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:39.324+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:39.674 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:39.325+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:39.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:39.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:39.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:39.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:39.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:39.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:39.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:39.310+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:39.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:39.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:39.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:39.900+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:40.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:40.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:40.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:40.314+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:40.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:40.288+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:40.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:40.288+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:40.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:40.288+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:40.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:40.289+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:41.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:40.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:41.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:40.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:41.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:40.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:41.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:40.870+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:41.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:41.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:41.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:41.265+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:41.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:41.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:41.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:41.326+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:41.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:41.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:41.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:41.837+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:42.364+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:42.364+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:42.364+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:42.365+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:42.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:42.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:42.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:42.217+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:42.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:42.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:42.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:42.791+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:43.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:43.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:43.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:43.380+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:43.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:43.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:43.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:43.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:43.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:43.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:43.223+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:44.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:43.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:44.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:43.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:44.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:43.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:44.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:43.766+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:44.077+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10155 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:44.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:44.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:44.192+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:44.193+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:44.775 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:44.418+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:44.775 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:44.418+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:44.775 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:44.418+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:44.775 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:44.419+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:44.772+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:44.772+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:44.772+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:44.773+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:45.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:45.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:45.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:45.194+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:45.414+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:45.414+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:45.414+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:45.414+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:45.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:45.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:45.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:45.757+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:46.462 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:46.162+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:46.462 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:46.162+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:46.463 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:46.162+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:46.463 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:46.163+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:46.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:46.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:46.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:46.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:46.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:46.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:46.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:46.460+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:46.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:46.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:46.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:46.798+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:47.454 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:47.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:47.455 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:47.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:47.455 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:47.180+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:47.455 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:47.181+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:47.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:47.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:47.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:47.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:47.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:47.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:47.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:47.453+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:47.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:47.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:47.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:47.799+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:48.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:48.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:48.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:48.425+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:48.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:48.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:48.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:48.194+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:49.096 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:49.078+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10207 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:49.096 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:48.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:49.097 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:48.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:49.097 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:48.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:49.097 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:48.837+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:49.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:49.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:49.419 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:49.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:49.419 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:49.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:49.419 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:49.211+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:49.416+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:49.416+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:49.416+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:49.417+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:50.207 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:49.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:50.207 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:49.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:50.207 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:49.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:50.207 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:49.809+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:50.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:50.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:50.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:50.390+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:50.204+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:50.204+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:50.204+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:50.205+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:50.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:50.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:50.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:50.804+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:51.351+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:51.351+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:51.351+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:51.352+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:51.174+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:51.174+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:51.174+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:51.175+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:52.143 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:51.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:52.143 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:51.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:52.144 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:51.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:52.144 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:51.827+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:52.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:52.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:52.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:52.384+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:52.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:52.141+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:52.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:52.141+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:52.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:52.141+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:52.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:52.141+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:53.117 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:52.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:53.117 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:52.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:53.117 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:52.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:53.118 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:52.854+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:53.376 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:53.115+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:53.377 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:53.115+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:53.377 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:53.115+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:53.377 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:53.115+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:53.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:53.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:53.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:53.375+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:54.082 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:53.818+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:54.082 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:53.818+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:54.082 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:53.818+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:54.082 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:53.819+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:54.417 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:54.079+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10257 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:54.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:54.132+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:54.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:54.132+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:54.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:54.132+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:54.418 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:54.134+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:54.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:54.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:54.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:54.416+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:55.110 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:54.815+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:55.111 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:54.815+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:55.111 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:54.815+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:55.111 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:54.815+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:55.455 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:55.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:55.455 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:55.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:55.455 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:55.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:55.455 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:55.109+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:55.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:55.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:55.452+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:55.453+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:56.064 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:55.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:56.064 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:55.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:56.064 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:55.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:56.064 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:55.812+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:56.064 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:56.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:56.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:56.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:56.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:56.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:56.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:56.061+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:56.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:56.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:56.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:56.431+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:57.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:57.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:57.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:57.076+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:56.843+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:56.843+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:56.843+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:56.845+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:57.468+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:57.468+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:57.468+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:57.469+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:58.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:57.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:58.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:57.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:58.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:57.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:58.129 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:57.877+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:58.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:58.127+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:58.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:58.127+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:58.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:58.127+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:58.498 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:58.128+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:58.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:58.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:58.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:58.496+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:59.169 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:42:59.080+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10308 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:42:59.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:58.884+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:59.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:58.884+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:59.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:58.884+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:42:59.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:58.885+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:42:59.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:59.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:59.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:59.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:59.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:59.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:42:59.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:42:59.094+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:42:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:59.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:59.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:59.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:42:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:42:59.490+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:59.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:59.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:59.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:42:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:42:59.901+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:00.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:00.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:00.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:00.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:00.084+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:00.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:00.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:00.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:00.457+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:01.178 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:01.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:01.178 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:01.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:01.178 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:01.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:01.179 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:01.051+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:01.179 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:00.915+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:01.179 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:00.915+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:01.179 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:00.915+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:01.179 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:00.916+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:01.475+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:01.475+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:01.475+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:01.476+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.027+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:02.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:01.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:01.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:01.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:01.947+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:02.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:02.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:02.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:02.452+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:02.984+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:02.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:02.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:02.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:02.912+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:03.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:03.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:03.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:03.463+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:04.205 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:04.081+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10358 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:04.205 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:03.970+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:04.205 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:03.970+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:04.206 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:03.970+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:04.206 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:03.971+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:04.206 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:03.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:04.206 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:03.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:04.206 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:03.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:04.206 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:03.951+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:04.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:04.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:04.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:04.471+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:04.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:04.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:04.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:04.943+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:04.986+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:04.986+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:04.986+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:04.987+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:05.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:05.434+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:05.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:05.434+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:05.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:05.434+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:05.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:05.435+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:05.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:05.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:05.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:05.945+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:05.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:05.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:05.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:05.968+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:06.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:06.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:06.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:06.455+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:06.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:06.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:06.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:06.944+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:06.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:06.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:06.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:06.990+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:07.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:07.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:07.458+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:07.458+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:07.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:07.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:07.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:07.999+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:07.947+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:07.947+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:07.947+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:07.949+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:08.411+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:08.411+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:08.411+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:08.411+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:09.246 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:09.082+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10409 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:09.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:08.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:09.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:08.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:09.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:08.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:09.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:08.942+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:09.247 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:08.955+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:09.247 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:08.955+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:09.247 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:08.955+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:09.247 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:08.955+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:09.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:09.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:09.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:09.415+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:09.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:09.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:09.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:09.980+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:09.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:09.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:09.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:09.899+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:10.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:10.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:10.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:10.402+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:11.014+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:11.014+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:11.014+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:11.014+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:10.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:11.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:10.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:11.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:10.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:11.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:10.892+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:11.366+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:11.366+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:11.366+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:11.367+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:11.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:11.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:11.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:11.876+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:12.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:12.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:12.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:12.041+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:12.337+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:12.337+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:12.337+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:12.338+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:13.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:13.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:13.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:13.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:13.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:13.032+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:13.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:13.032+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:13.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:12.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:13.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:12.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:13.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:12.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:13.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:12.846+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:13.294+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:13.294+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:13.294+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:13.295+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:14.284 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:14.083+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10459 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:14.284 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:13.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:14.284 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:13.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:14.285 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:13.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:14.285 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:13.837+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:14.285 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:14.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:14.285 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:14.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:14.285 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:14.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:14.285 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:14.021+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:14.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:14.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:14.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:14.301+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:15.305 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:15.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:15.305 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:15.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:15.305 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:15.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:15.305 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:15.020+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:15.305 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:14.812+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:15.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:14.812+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:15.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:14.812+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:15.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:14.813+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:15.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:15.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:15.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:15.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:15.303+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:16.055 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:15.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:16.056 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:15.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:16.056 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:15.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:16.056 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:15.801+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:16.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:16.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:16.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:16.296+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:16.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:16.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:16.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:16.054+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:17.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:17.015+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:17.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:17.015+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:17.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:17.015+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:17.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:17.016+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:17.249 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:16.832+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:17.249 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:16.832+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:17.250 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:16.832+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:17.250 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:16.833+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:17.246+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:17.246+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:17.246+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:17.247+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:18.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:18.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:18.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:18.022+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:17.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:17.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:17.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:17.796+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:18.268+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:18.268+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:18.268+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:18.269+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:19.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:19.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:19.016+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:19.016+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:19.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:18.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:19.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:18.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:19.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:18.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:19.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:18.809+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:19.084+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10510 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:19.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:19.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:19.285+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:19.286+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:20.025 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:19.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:20.025 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:19.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:20.025 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:19.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:20.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:19.805+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:20.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:20.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:20.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:20.268+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.023+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:20.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:20.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:20.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:20.761+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.987+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.987+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.987+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:20.988+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:21.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:21.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:21.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:21.283+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:22.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:21.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:22.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:21.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:22.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:21.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:22.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:21.787+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:22.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:22.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:22.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:22.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:22.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:22.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:22.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:22.031+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:22.297+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:22.297+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:22.297+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:22.298+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:23.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:22.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:23.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:22.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:23.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:22.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:23.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:22.801+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.023+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.023+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.023+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.024+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:23.583 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:23.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:23.583 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:23.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:23.583 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:23.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:23.583 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:23.310+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:24.088 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:23.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:24.088 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:23.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:24.088 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:23.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:24.089 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:23.832+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:24.089 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:24.089 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:24.089 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:24.089 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:23.980+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:24.368 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:24.085+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10560 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:24.369 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:24.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:24.369 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:24.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:24.369 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:24.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:24.369 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:24.270+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:24.816+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:24.816+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:24.816+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:24.817+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:25.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:25.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:25.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:25.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:25.004+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:25.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:25.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:25.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:25.243+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:26.038 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:25.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:26.039 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:25.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:26.039 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:25.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:26.039 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:25.853+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.037+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:26.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:26.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:26.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:26.229+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:26.991+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:26.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:26.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:26.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:26.808+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:27.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:27.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:27.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:27.197+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:27.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:27.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:27.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:27.969+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:27.790+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:27.790+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:27.790+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:27.791+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:28.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:28.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:28.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:28.208+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:28.942+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:28.942+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:28.942+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:28.943+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:28.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:28.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:29.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:28.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:29.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:28.762+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:29.407 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:29.086+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10611 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:29.407 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:29.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:29.408 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:29.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:29.408 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:29.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:29.408 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:29.227+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:29.992 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:29.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:29.992 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:29.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:29.992 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:29.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:29.992 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:29.734+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:30.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:30.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:30.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:30.185+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:29.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:29.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:29.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:29.990+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:31.021 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:30.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:31.021 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:30.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:31.021 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:30.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:31.022 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:30.766+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:31.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:31.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:31.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:31.185+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:31.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:31.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:31.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:31.019+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:31.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:31.020+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:32.036 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:31.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:32.036 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:31.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:32.037 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:31.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:32.037 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:31.779+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:32.232+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:32.232+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:32.232+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:32.233+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:32.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:32.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:32.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:32.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:32.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:32.035+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:33.216 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:33.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:33.217 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:33.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:33.217 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:33.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:33.217 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:33.056+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:33.217 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:32.818+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:33.217 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:32.818+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:33.217 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:32.818+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:33.217 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:32.818+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:33.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:33.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:33.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:33.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:33.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:33.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:33.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:33.215+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:34.090 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:34.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:34.091 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:34.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:34.091 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:34.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:34.091 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:34.026+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:34.091 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:33.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:34.091 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:33.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:34.091 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:33.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:34.091 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:33.834+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:34.446 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:34.087+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10661 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:34.447 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:34.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:34.447 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:34.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:34.447 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:34.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:34.447 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:34.201+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:35.186 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:35.065+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:35.187 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:35.065+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:35.187 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:35.065+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:35.187 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:35.066+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:35.187 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:34.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:35.187 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:34.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:35.187 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:34.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:35.187 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:34.881+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:35.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:35.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:35.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:35.185+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:36.203 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:36.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:36.203 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:36.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:36.203 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:36.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:36.203 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:36.020+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:36.204 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:35.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:36.204 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:35.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:36.204 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:35.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:36.204 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:35.841+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:36.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:36.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:36.200+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:36.201+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:36.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:36.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:36.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:37.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:36.799+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:37.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:37.011+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:37.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:37.011+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:37.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:37.011+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:37.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:37.012+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:37.530 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:37.230+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:37.530 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:37.230+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:37.530 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:37.230+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:37.530 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:37.231+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:38.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:37.787+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:38.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:37.787+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:38.048 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:37.787+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:38.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:37.788+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:38.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:38.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:38.254+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:38.255+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:38.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:38.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:38.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:38.046+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:39.091 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:39.092 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:39.092 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:39.092 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.038+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:39.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:38.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:39.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:38.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:39.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:38.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:39.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:38.827+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:39.484 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:39.089+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10712 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:39.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:39.245+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:39.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:39.245+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:39.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:39.245+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:39.484 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:39.246+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:40.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:39.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:40.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:39.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:40.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:39.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:40.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:39.839+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:40.248 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:40.248 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:40.248 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:40.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:39.997+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:40.245+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:40.245+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:40.245+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:40.246+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:41.114 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:41.044+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:41.114 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:41.044+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:41.114 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:41.044+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:41.115 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:41.045+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:41.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:40.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:41.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:40.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:41.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:40.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:41.115 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:40.834+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:41.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:41.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:41.197+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:41.198+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:42.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:42.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:42.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:42.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:42.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:42.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:42.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:42.064+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:41.870+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:41.870+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:41.870+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:42.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:41.871+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:42.215+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:42.215+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:42.215+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:42.216+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:43.246 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:43.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:43.246 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:43.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:43.246 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:43.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:43.246 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:43.039+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:43.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:42.878+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:43.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:42.878+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:43.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:42.878+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:43.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:42.879+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:43.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:43.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:43.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:43.244+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:44.187 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:44.089+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10762 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:44.187 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:44.081+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:44.187 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:44.081+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:44.187 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:44.081+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:44.187 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:44.082+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:44.188 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:43.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:44.188 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:43.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:44.188 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:43.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:44.188 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:43.872+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:44.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:44.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:44.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:44.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:44.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:44.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:44.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:44.223+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:45.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:45.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:45.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:45.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:45.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:45.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:45.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:45.086+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:45.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:44.912+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:45.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:44.912+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:45.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:44.912+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:45.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:44.912+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:45.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:45.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:45.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:45.253+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:46.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:46.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:46.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:46.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.041+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:46.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:45.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:46.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:45.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:46.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:45.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:46.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:45.930+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:46.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:46.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:46.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:46.216+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:47.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:47.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:47.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.995+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:47.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:46.996+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:47.225 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:46.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:47.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:46.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:47.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:46.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:47.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:46.884+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:47.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:47.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:47.222+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:47.223+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:47.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:47.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:47.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:47.880+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:48.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:48.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:48.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:48.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:48.004+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:48.212+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:48.212+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:48.212+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:48.212+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:49.176 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:49.091+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10813 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:49.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:49.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:49.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:49.022+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:48.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:48.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:48.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:49.177 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:48.922+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:49.558 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:49.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:49.558 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:49.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:49.559 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:49.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:49.559 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:49.175+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:49.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:49.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:49.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:49.949+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:50.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:50.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:50.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:50.175+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:50.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:50.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:50.026+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:50.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:50.027+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:51.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:51.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:51.068+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:51.069+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:50.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:50.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:50.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:50.976+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:51.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:51.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:51.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:51.186+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:52.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:52.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:52.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:52.185+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:51.952+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:51.952+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:51.952+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:51.952+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:52.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:52.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:52.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:52.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:52.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:52.055+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:52.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:52.056+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:53.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:53.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:53.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:53.155+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:53.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:53.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:53.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:53.008+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:52.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:52.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:52.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:52.926+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:54.154 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:54.092+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10863 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:54.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:54.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:54.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:54.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:54.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:54.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:54.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:54.006+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:54.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:53.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:54.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:53.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:54.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:53.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:54.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:53.884+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:54.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:54.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:54.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:54.152+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:55.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:55.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:55.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:55.161+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:55.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:55.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:55.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:55.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:55.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:55.005+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:54.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:54.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:54.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:54.932+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:56.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:56.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:56.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:56.118+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:56.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.028+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:55.904+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:55.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:55.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:55.905+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:57.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.997+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:57.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.997+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:57.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.997+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:57.149 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:56.999+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:57.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:56.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:57.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:56.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:57.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:56.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:57.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:56.868+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:57.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:57.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:57.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:57.147+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:57.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:58.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:57.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:58.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:57.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:58.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:57.996+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:58.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:57.912+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:58.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:57.912+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:58.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:57.912+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:58.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:57.912+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:58.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:58.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:58.314 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:58.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:58.314 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:58.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:58.314 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:58.154+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:59.166 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:43:59.092+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10915 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:59.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:59.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:59.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:43:59.012+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:58.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:58.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:58.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:43:59.167 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:58.867+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:43:59.545 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:59.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:59.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:59.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:59.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:59.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:43:59.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:43:59.164+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:00.180 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:00.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:00.180 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:00.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:00.180 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:00.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:00.180 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:00.008+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:00.180 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:59.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:00.180 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:59.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:00.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:59.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:00.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:43:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:43:59.869+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:00.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:00.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:00.176+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:00.177+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:01.209 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:01.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:01.210 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:01.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:01.210 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:01.051+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:01.210 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:01.052+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:01.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:00.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:01.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:00.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:01.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:00.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:01.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:00.902+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:01.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:01.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:01.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:01.208+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:02.226 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:02.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:02.226 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:02.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:02.226 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:02.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:02.226 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:02.011+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:02.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:01.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:02.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:01.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:02.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:01.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:02.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:01.935+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:02.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:02.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:02.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:02.224+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:03.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:02.940+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:03.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:02.940+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:03.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:02.940+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:03.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:02.940+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:03.226 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:03.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:03.226 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:03.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:03.227 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:03.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:03.227 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:03.030+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:03.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:03.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:03.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:03.224+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:04.254 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:04.094+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 10966 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:04.254 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:04.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:04.254 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:04.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:04.254 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:04.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:04.254 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:04.061+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:04.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:03.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:04.255 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:03.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:04.255 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:03.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:04.255 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:03.930+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:04.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:04.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:04.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:04.252+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:05.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:04.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:05.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:04.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:05.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:04.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:05.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:04.914+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:05.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:05.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:05.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:05.275+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:05.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:05.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:05.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:05.047+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:05.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:05.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:05.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:05.867+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:06.097+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:06.097+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:06.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:06.097+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:06.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:06.098+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:06.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:06.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:06.310+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:06.311+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:07.264 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:07.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:07.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:07.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:07.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:07.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:07.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:06.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:07.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:06.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:07.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:06.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:07.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:06.914+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:07.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:07.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:07.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:07.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:07.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:07.059+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:07.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:07.060+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:07.263+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:08.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:08.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:08.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:08.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:08.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:08.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:08.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:08.097+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:08.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:07.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:08.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:07.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:08.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:07.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:08.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:07.922+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:08.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:08.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:08.242+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:08.243+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:08.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:08.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:08.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:08.881+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:09.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:09.095+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11018 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:09.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:09.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:09.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:09.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:09.257+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:09.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:09.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:09.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:09.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:09.064+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:10.046 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:09.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:10.046 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:09.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:10.046 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:09.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:10.046 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:09.851+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:10.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:10.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:10.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:10.256+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:10.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:10.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:10.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:10.044+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:11.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:11.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:11.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:11.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:11.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:11.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:11.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:11.005+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:11.307 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:10.874+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:11.307 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:10.874+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:11.307 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:10.874+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:11.307 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:10.875+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:11.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:11.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:11.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:11.304+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:12.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:12.047+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:12.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:12.047+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:12.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:12.047+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:12.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:12.048+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:12.302 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:11.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:12.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:11.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:12.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:11.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:12.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:11.892+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:12.299+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:12.299+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:12.299+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:12.300+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:13.261 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:12.855+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:13.261 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:12.855+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:13.261 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:12.855+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:13.261 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:12.855+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:13.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:13.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:13.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:13.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:13.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:13.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:13.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:13.034+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:13.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:13.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:13.258+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:13.259+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:13.829+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:13.829+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:13.829+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:13.830+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:14.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:14.096+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11070 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:14.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:14.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:14.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:14.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:14.259+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:14.073+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:14.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:14.073+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:14.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:14.073+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:14.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:14.074+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:14.799+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:14.799+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:14.799+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:14.800+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:15.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:15.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:15.270+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:15.270+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:15.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:15.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:15.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:15.113+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:15.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:15.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:15.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:15.793+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:16.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:16.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:16.251+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:16.252+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:16.152+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:16.152+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:16.152+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:16.153+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:16.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:16.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:16.803+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:16.803+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:17.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:17.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:17.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:17.253+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:17.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:17.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:17.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:17.119+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:18.232 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:18.074+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:18.232 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:18.074+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:18.232 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:18.074+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:18.232 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:18.074+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:18.232 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:17.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:18.233 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:17.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:18.233 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:17.834+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:18.233 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:17.834+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:18.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:18.229+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:18.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:18.229+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:18.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:18.229+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:18.230+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:19.101 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:18.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:19.101 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:18.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:19.101 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:18.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:19.101 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:18.840+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:19.102 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:19.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:19.102 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:19.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:19.102 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:19.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:19.102 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:19.078+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:19.097+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11121 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:19.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:19.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:19.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:19.218+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:19.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:19.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:19.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:19.841+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:20.201+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:20.201+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:20.201+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:20.201+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:20.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:20.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:20.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:20.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:20.059+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:20.886+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:20.886+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:20.886+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:20.886+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:21.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:21.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:21.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:21.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:21.054+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:21.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:21.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:21.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:21.229+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:22.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:22.049+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:22.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:22.049+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:22.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:22.049+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:22.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:22.049+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:22.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:21.882+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:22.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:21.882+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:22.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:21.882+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:22.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:21.883+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:22.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:22.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:22.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:22.275+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:23.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:23.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:23.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:23.046+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:22.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:22.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:22.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:22.906+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:23.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:23.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:23.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:23.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:23.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:23.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:23.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:23.312+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:24.098+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11172 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:24.072+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:24.072+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:24.072+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:24.073+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:23.893+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:23.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:23.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:23.894+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:24.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:24.342+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:24.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:24.342+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:24.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:24.342+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:24.634 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:24.343+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:25.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:25.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:25.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:25.075+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:24.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:24.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:24.867+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:25.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:24.868+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:25.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:25.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:25.303+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:25.304+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:25.902+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:25.902+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:25.902+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:25.902+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:26.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:26.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:26.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:26.047+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:26.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:26.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:26.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:26.323+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:27.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:26.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:27.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:26.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:27.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:26.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:26.930+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.011+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.011+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.011+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.011+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:27.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:27.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:27.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:27.308+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:28.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:28.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:28.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:28.287 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:27.969+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:28.288 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:27.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:28.288 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:27.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:28.288 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:27.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:28.288 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:27.931+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:28.284+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:28.284+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:28.284+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:28.285+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:29.285 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:29.099+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11223 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:29.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:29.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:29.018+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:29.019+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:28.945+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:28.945+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:28.945+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:29.286 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:28.945+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:29.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:29.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:29.282+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:29.283+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:30.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:30.042+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:30.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:30.042+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:30.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:30.042+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:30.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:30.043+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:30.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:29.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:30.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:29.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:30.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:29.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:30.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:29.939+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:30.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:30.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:30.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:30.296+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:31.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:31.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:31.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:31.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.004+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:31.277 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:30.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:31.277 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:30.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:31.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:30.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:31.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:30.930+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:31.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:31.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:31.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:31.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:31.275+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:31.954+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:31.954+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:31.954+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:31.954+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:32.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:31.958+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:32.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:32.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:32.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:32.301+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:32.914+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:32.914+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:32.914+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:32.916+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:32.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:32.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:32.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:32.972+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:33.588 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:33.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:33.588 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:33.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:33.588 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:33.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:33.589 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:33.315+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:34.100+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11273 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:33.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:33.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:33.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:33.884+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:33.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:33.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:33.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:33.970+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:34.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:34.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:34.323+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:34.324+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:35.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:34.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:35.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:34.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:35.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:34.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:35.278 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:34.984+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:35.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:34.885+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:35.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:34.885+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:35.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:34.885+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:35.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:34.886+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:35.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:35.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:35.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:35.276+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:36.230 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:35.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:36.230 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:35.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:36.230 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:35.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:36.230 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:35.995+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:36.230 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:35.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:36.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:35.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:36.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:35.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:36.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:35.910+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:36.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:36.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:36.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:36.228+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:37.279 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:37.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:37.279 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:37.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:37.279 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:37.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:37.280 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:37.040+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:37.280 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:36.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:37.280 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:36.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:37.280 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:36.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:37.280 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:36.932+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:37.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:37.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:37.277+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:37.277+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:38.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:38.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:38.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:38.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:38.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:38.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:38.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:38.027+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:38.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:37.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:38.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:37.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:38.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:37.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:38.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:37.936+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:38.292+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:38.292+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:38.292+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:38.293+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:39.278 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:39.102+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11323 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:38.972+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:38.972+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:38.972+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:38.973+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.010+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.010+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.010+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:39.279 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.011+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:39.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:39.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:39.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:39.277+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:40.256 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:40.256 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:40.256 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.005+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:40.256 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.005+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:40.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:40.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:40.256 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:40.257 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:39.998+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:40.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:40.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:40.253+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:40.254+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:40.973+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:40.973+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:40.973+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:40.974+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:41.206+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:41.206+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:41.206+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:41.206+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:40.995+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:42.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:41.935+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:42.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:41.935+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:42.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:41.935+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:42.215 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:41.935+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:42.216 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:42.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:42.216 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:42.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:42.216 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:42.045+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:42.216 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:42.046+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:42.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:42.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:42.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:42.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:42.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:42.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:42.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:42.213+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:43.259 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:43.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:43.259 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:43.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:43.259 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:43.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:43.260 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:43.002+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:43.260 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:42.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:43.260 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:42.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:43.260 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:42.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:43.260 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:42.895+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:43.256+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:43.256+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:43.256+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:43.257+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:44.235 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:44.103+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11372 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:44.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:44.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:44.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.035+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:44.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.035+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:44.235 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:43.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:44.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:43.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:44.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:43.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:44.236 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:43.884+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:44.232+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:44.232+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:44.232+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:44.233+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:45.274 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:45.274 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:45.274 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:45.275 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:44.990+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:45.275 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:44.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:45.275 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:44.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:45.275 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:44.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:45.275 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:44.906+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:45.271+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:45.271+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:45.271+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:45.272+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:46.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:46.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:46.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:46.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:46.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:46.038+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:46.263 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:46.039+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:46.263 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:45.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:46.263 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:45.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:46.263 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:45.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:46.263 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:45.893+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:46.259+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:46.259+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:46.259+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:46.260+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:46.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:46.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:46.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:46.901+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:47.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:47.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:47.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:47.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:47.034+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:47.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:47.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:47.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:47.301+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.010+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.010+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:47.887+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:47.887+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:47.887+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:47.888+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:48.698 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:48.341+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:48.699 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:48.341+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:48.699 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:48.341+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:48.699 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:48.342+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:49.103+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11423 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:48.983+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:48.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:48.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:48.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:48.938+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:49.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:49.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:49.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:49.341+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:49.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:49.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:49.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:49.992+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:49.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:49.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:49.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:50.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:49.923+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:50.368+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:50.368+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:50.368+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:50.368+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:50.937+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:50.937+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:50.937+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:50.938+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:51.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:51.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:51.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:51.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.008+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:51.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:51.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:51.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:51.416+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:51.961+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:51.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:51.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:51.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:51.906+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:52.436+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:52.436+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:52.436+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:52.437+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:52.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:52.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:52.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:52.949+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:52.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:52.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:52.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:53.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:52.878+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:53.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:53.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:53.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:53.454+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:54.105+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11473 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:53.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:53.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:53.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:54.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:53.905+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:53.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:53.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:53.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:53.862+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:54.413+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:54.413+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:54.413+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:54.414+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:54.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:54.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:54.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:54.938+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:54.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:54.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:54.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:55.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:54.878+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:55.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:55.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:55.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:55.446+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:55.950+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:55.950+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:55.950+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:55.950+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:55.926+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:55.926+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:55.926+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:55.927+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:56.413+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:56.413+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:56.413+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:56.413+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:56.966+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:56.966+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:56.966+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:56.967+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:56.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:56.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:56.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:56.933+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:57.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:57.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:57.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:57.402+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:58.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:57.985+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:58.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:57.985+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:58.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:57.985+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:58.154 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:57.986+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:58.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:57.890+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:58.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:57.890+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:58.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:57.890+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:58.155 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:57.890+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:58.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:58.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:58.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:58.432+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:59.109 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:58.952+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:59.109 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:58.952+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:59.109 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:58.952+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:44:59.109 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:58.953+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:44:59.109 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:58.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:59.109 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:58.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:59.110 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:58.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:44:59.110 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:58.851+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:44:59.447 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:44:59.106+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11524 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:44:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:59.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:59.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:59.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:44:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:44:59.445+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:59.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:59.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:59.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:44:59.910+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:59.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:00.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:59.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:00.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:59.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:00.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:44:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:44:59.871+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:00.423+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:00.423+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:00.423+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:00.424+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:00.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:00.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:00.909+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:00.910+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:01.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:00.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:01.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:00.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:01.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:00.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:01.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:00.889+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:01.380+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:01.380+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:01.380+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:01.381+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:01.947+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:01.947+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:01.947+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:01.948+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:02.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:01.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:02.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:01.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:01.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:01.914+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:02.371+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:02.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:02.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:02.373+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:02.982+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:02.982+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:02.982+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:02.983+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:02.936+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:02.936+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:02.936+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:02.937+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:03.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:03.408+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:03.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:03.408+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:03.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:03.408+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:03.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:03.409+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:04.107+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11574 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:03.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:03.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:03.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:03.957+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:04.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:03.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:04.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:03.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:04.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:03.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:04.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:03.965+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:04.447+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:04.447+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:04.447+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:04.448+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:05.000+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:05.000+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:05.000+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:05.001+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:04.970+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:04.970+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:04.970+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:04.970+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:05.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:05.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:05.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:05.479+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:05.961+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:05.961+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:05.961+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:05.961+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:06.033+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:06.033+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:06.033+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:06.034+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:06.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:06.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:06.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:06.527+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:07.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:45:07.203+0000 7f03ba68c700 0 monclient(hunting): authenticate timed out after 300 2024-03-12T03:45:07.207 INFO:teuthology.orchestra.run.smithi118.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T03:45:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:06.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:06.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:06.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:06.957+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:07.047+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:07.047+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:07.047+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:07.048+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:07.578 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:07.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:07.578 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:07.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:07.578 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:07.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:07.578 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:07.489+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:07.978 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-12T03:45:08.208 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/mon.smithi118/config 2024-03-12T03:45:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:08.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:08.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:08.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:08.081+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:08.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:08.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:08.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:08.003+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:08.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:08.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:08.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:08.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:08.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:08.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:08.587 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:08.519+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:09.108+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11624 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:09.031+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:09.031+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:09.031+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:09.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:09.032+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:09.090+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:09.090+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:09.090+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:09.091+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:09.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:09.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:09.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:09.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:09.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:09.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:09.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:09.543+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:10.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:10.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:10.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:10.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:10.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:10.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:10.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:10.073+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:10.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:10.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:10.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:10.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:10.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:10.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:10.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:10.064+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:10.540+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:10.540+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:10.540+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:10.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:10.541+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:11.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:11.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:11.149 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:11.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:11.149 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:11.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:11.149 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:11.086+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:11.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:11.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:11.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:11.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:11.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:11.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:11.149 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:11.062+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:11.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:11.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:11.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:11.506+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:12.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:12.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:12.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:12.063+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:12.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:12.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:12.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:12.031+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:12.474+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:12.474+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:12.474+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:12.475+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:13.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:13.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:13.086+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:13.086+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:13.022+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:13.022+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:13.022+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:13.022+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:13.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:13.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:13.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:13.495+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:14.110+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11673 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:14.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:14.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:14.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:14.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:14.118+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:14.004+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:14.004+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:14.004+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:14.005+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:14.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:14.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:14.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:14.508+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:15.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:15.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:15.107+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:15.107+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.002+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:15.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:15.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:15.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:15.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:15.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:15.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:15.520+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:16.100+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:16.100+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:16.100+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:16.101+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.973+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.973+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.973+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:16.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:15.974+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:16.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:16.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:16.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:16.532+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:17.105+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:17.105+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:17.105+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:17.106+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:16.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:16.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:16.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:16.976+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:17.960 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:17.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:17.960 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:17.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:17.960 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:17.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:17.960 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:17.568+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:18.132+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:18.132+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:18.132+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:18.132+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:17.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:17.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:17.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:17.959+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:18.971 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:18.561+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:18.971 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:18.561+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:18.972 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:18.561+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:18.972 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:18.561+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:19.111+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11722 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:19.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:19.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:19.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:19.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:19.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:19.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:19.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:19.120+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:18.968+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:18.968+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:18.968+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:18.969+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:19.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:19.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:19.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:19.526+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:20.257 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:20.152+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:20.257 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:20.152+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:20.257 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:20.152+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:20.257 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:20.153+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:20.257 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:19.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:20.257 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:19.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:20.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:19.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:20.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:19.932+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:20.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:20.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:20.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:20.478+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:21.114+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:21.114+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:21.114+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:21.114+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:20.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:20.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:20.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:20.894+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:21.574 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:21.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:21.575 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:21.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:21.575 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:21.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:21.575 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:21.494+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:22.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:22.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:22.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:22.088+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:21.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:21.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:21.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:22.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:21.860+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:22.482+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:22.482+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:22.482+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:22.483+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:23.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:22.855+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:23.124 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:22.855+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:23.124 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:22.855+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:23.124 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:22.855+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:23.489 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:23.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:23.489 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:23.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:23.489 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:23.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:23.489 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:23.122+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:23.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:23.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:23.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:23.487+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:24.128 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:24.112+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11772 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:24.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:23.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:24.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:23.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:24.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:23.871+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:24.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:23.872+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:24.534 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:24.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:24.534 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:24.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:24.534 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:24.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:24.534 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:24.126+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:24.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:24.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:24.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:24.532+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:25.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:25.147+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:25.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:25.147+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:25.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:25.147+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:25.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:25.148+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:25.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:24.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:25.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:24.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:25.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:24.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:25.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:24.920+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:25.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:25.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:25.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:25.543+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:26.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:26.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:26.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:26.154+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:25.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:25.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:25.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:25.958+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:26.951 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:26.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:26.951 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:26.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:26.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:26.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:26.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:26.568+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:27.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:27.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:27.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:27.179+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:26.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:26.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:26.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:26.950+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:27.928 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:27.602+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:27.929 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:27.602+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:27.929 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:27.602+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:27.929 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:27.603+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:28.216 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:27.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:28.216 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:27.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:28.216 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:27.925+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:28.216 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:27.926+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:28.213+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:28.213+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:28.213+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:28.214+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:28.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:28.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:28.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:28.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:28.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:28.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:28.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:28.619+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:29.267 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:29.113+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11821 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:29.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:28.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:29.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:28.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:29.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:28.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:29.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:28.944+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:29.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:29.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:29.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:29.265+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:29.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:29.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:29.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:29.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:29.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:29.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:29.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:29.663+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:30.240 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:29.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:30.240 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:29.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:30.240 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:29.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:30.241 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:29.945+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:30.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:30.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:30.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:30.238+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:30.932 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:30.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:30.932 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:30.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:30.932 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:30.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:30.932 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:30.670+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:31.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:30.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:31.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:30.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:31.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:30.929+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:31.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:30.930+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:31.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:31.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:31.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:31.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:31.964 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:31.665+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:31.964 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:31.665+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:31.964 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:31.665+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:31.964 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:31.666+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:32.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:31.961+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:32.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:31.961+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:32.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:31.961+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:32.283 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:31.962+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:32.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:32.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:32.280+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:32.281+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:32.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:32.694+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:32.694+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:32.694+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:32.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:32.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:32.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:33.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:32.917+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:33.593 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:33.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:33.593 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:33.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:33.593 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:33.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:33.593 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:33.331+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:33.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:33.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:33.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:33.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:33.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:33.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:33.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:33.711+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:34.114+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11872 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:33.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:33.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:33.944+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:33.945+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:34.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:34.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:34.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:34.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:34.694 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:34.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:34.694 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:34.361+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:34.690+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:34.690+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:34.690+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:34.691+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:34.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:34.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:34.899+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:34.900+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:35.651 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:35.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:35.652 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:35.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:35.652 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:35.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:35.652 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:35.344+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:35.939 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:35.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:35.939 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:35.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:35.940 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:35.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:35.940 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:35.650+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:35.937+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:35.937+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:35.937+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:35.937+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:36.666 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:36.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:36.666 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:36.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:36.666 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:36.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:36.666 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:36.377+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:36.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:36.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:36.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:36.663+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:36.891+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:36.891+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:36.891+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:37.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:36.891+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:37.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:37.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:37.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:37.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:37.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:37.361+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:37.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:37.361+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:37.922 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:37.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:37.922 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:37.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:37.922 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:37.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:37.922 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:37.631+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:37.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:37.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:37.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:37.920+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:38.684 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:38.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:38.684 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:38.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:38.684 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:38.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:38.684 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:38.359+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:38.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:38.681+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:38.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:38.681+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:38.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:38.681+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:38.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:38.682+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:39.115+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11921 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:38.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:38.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:38.943+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:38.943+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:39.657 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:39.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:39.657 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:39.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:39.657 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:39.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:39.657 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:39.333+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:39.958 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:39.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:39.958 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:39.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:39.958 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:39.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:39.959 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:39.655+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:39.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:39.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:39.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:39.956+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:40.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:40.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:40.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:40.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:40.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:40.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:40.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:40.358+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:40.998 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:40.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:40.998 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:40.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:40.998 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:40.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:40.998 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:40.679+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:40.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:40.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:40.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:40.993+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:41.659 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:41.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:41.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:41.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:41.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:41.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:41.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:41.402+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:41.660 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:41.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:41.660 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:41.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:41.660 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:41.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:41.999 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:41.658+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:41.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:41.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:41.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:41.998+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:42.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:42.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:42.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:42.644+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:42.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:42.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:42.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:42.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:42.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:42.415+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:42.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:42.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:42.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:42.973+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:43.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:43.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:43.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:43.631+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:43.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:43.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:43.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:43.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:43.466+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:44.116+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 11972 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.015+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.015+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.015+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.016+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:44.628 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:44.508+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:44.628 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:44.508+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:44.628 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:44.508+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:44.628 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:44.509+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:45.000 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:44.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:45.000 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:44.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:45.000 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:44.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:45.000 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:44.626+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:44.998+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:45.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:45.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:45.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:45.548+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:45.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:45.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:45.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:45.634+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.039+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.039+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:46.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:46.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:46.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:46.540+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:46.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:46.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:46.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:46.674+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.993+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.993+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.993+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:46.994+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:47.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:47.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:47.814 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:47.511+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:47.814 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:47.511+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:47.814 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:47.512+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:47.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:47.688+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:47.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:47.688+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:47.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:47.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:47.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:47.689+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:47.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:47.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:47.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:47.950+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:48.493+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:48.493+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:48.493+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:48.494+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:48.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:48.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:48.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:48.675+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:49.118+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12021 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:48.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:48.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:48.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:48.916+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:49.533+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:49.533+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:49.533+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:49.533+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:49.650+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:49.650+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:49.650+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:49.650+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:49.965+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:49.965+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:49.965+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:49.965+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:50.527+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:50.527+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:50.527+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:50.529+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:50.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:50.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:50.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:50.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:50.606+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:50.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:50.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:50.942+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:50.943+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:51.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:51.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:51.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:51.643+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:51.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:51.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:51.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:51.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:51.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:51.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:51.502+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:51.977+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:51.977+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:51.977+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:51.978+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:52.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:52.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:52.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:52.684+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:52.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:52.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:52.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:52.484+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.002+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:53.641+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:53.641+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:53.641+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:53.642+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:53.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:53.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:53.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:53.451+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:54.119+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12072 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.985+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.985+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.985+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:53.986+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:54.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:54.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:54.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:54.409+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:54.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:54.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:54.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:54.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:54.604+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:54.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:54.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:54.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:54.981+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:55.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:55.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:55.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:55.371+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:55.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:55.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:55.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:55.599+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:56.281 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:56.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:56.282 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:56.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:56.282 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:56.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:21.828870+0000 (oldest deadline 2024-03-12T03:35:41.828870+0000) 2024-03-12T03:45:56.282 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:56.027+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:56.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:56.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:56.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:56.366+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:57.038 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:56.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:57.038 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:56.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:57.038 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:56.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:57.038 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:56.605+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:57.035+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:57.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:57.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:57.615 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:57.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:57.615 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:57.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:57.615 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:57.336+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:57.999 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:57.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:58.000 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:57.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:58.000 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:57.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:58.000 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:57.613+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:58.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:57.997+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:58.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:58.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:58.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:58.569+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:58.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:58.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:58.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:58.351+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:45:59.120+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12122 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:45:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:58.981+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:45:59.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:59.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:59.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:59.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:59.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:59.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:45:59.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:45:59.337+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:45:59.942 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:59.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:59.942 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:59.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:59.943 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:59.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:45:59.943 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:45:59.616+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:45:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:45:59.940+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:00.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:00.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:00.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:00.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:00.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:00.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:00.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:00.331+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:00.957 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:00.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:00.957 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:00.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:00.957 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:00.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:00.957 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:00.612+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:01.308 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:00.954+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:01.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:01.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:01.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:01.306+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:01.940 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:01.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:01.941 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:01.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:01.941 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:01.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:01.941 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:01.612+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:02.308 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:01.937+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:02.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:02.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:02.306+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:02.306+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:02.905 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:02.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:02.906 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:02.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:02.906 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:02.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:02.906 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:02.621+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:02.903+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:03.595 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:03.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:03.595 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:03.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:03.595 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:03.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:03.595 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:03.330+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:03.595 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:03.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:03.596 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:03.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:03.596 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:03.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:03.596 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:03.576+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:04.121+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12173 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:03.876+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:04.625 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:04.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:04.625 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:04.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:04.625 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:04.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:04.625 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:04.351+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:04.903 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:04.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:04.904 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:04.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:04.904 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:04.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:04.904 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:04.623+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:05.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:04.901+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:05.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:05.331+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:05.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:05.331+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:05.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:05.331+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:05.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:05.331+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:05.950 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:05.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:05.950 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:05.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:05.950 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:05.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:05.950 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:05.641+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:06.307 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:05.947+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:06.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:06.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:06.305+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:06.306+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:06.961 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:06.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:06.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:06.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:06.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:06.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:06.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:06.633+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:06.959+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:07.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:07.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:07.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:07.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:07.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:07.346+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:07.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:07.346+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:07.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:07.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:07.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:07.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:07.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:07.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:07.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:07.646+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:07.923+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:08.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:08.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:08.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:08.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:08.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:08.365+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:08.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:08.366+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:08.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:08.657+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:08.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:08.657+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:08.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:08.657+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:31.346217+0000 (oldest deadline 2024-03-12T03:35:51.346217+0000) 2024-03-12T03:46:08.926 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:08.659+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:09.123+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12222 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:08.923+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:09.675 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:09.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:09.675 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:09.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:09.675 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:09.326+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:09.676 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:09.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:09.961 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:09.672+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:09.958+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:10.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:10.352+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:10.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:10.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:10.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:10.353+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:10.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:10.353+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:10.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:10.693+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:10.973+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:11.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:11.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:11.642 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:11.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:11.642 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:11.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:35:40.473439+0000 (oldest deadline 2024-03-12T03:36:00.473439+0000) 2024-03-12T03:46:11.642 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:11.370+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:11.986 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:11.708+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:12.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:11.984+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:12.666 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:12.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:12.972+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:12.759+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:13.595 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:13.315+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:14.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:13.785+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:14.299 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:14.124+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12272 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:14.299 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:14.012+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:14.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:14.296+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:14.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:14.791+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:15.276 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:14.972+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:15.274+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:15.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:15.824+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:15.994+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:16.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:16.312+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:16.800+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:17.019+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:17.786 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:17.343+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:17.844+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:17.990+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:18.324+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:19.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:18.812+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:19.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:19.022+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:19.125+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12321 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:19.304+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:20.299 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:20.006+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:20.300 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:19.820+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:20.297+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:20.962+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:20.795+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:21.774 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:21.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:21.772+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:21.965+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:21.965+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:21.965+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:21.966+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:22.758 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:22.372+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:22.756+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:22.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:22.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:22.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:22.989+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:23.710 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:23.366+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:23.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:23.707+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:24.126+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12371 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:23.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:23.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:23.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:23.994+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:24.729 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:24.408+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:25.002 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:24.725+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:24.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:24.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:24.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:25.000+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:25.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:25.401+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:26.052 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:25.750+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:26.050+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:26.050+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:26.050+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:26.050+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:26.714 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:26.388+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:27.033 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:26.712+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:27.030+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:27.030+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:27.030+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:27.031+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:27.733 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:27.369+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:27.731+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:28.354 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:28.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:28.354 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:28.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:28.354 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:28.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:28.355 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:28.073+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:28.730 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:28.352+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:29.033 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:28.727+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:29.127+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12420 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:29.030+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:29.030+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:29.030+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:29.031+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:29.779 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:29.364+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:29.777+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:30.334 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:30.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:30.335 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:30.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:30.335 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:30.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:30.335 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:30.071+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:30.755 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:30.332+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:31.036 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:30.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:31.036 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:30.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:31.037 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:30.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:31.037 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:30.753+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:31.034+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:31.034+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:31.034+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:31.034+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:31.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:31.354+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:32.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:31.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:32.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:31.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:32.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:31.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:32.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:31.738+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:32.056+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:32.056+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:32.056+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:32.058+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:32.784 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:32.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:32.784 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:32.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:32.784 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:32.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:32.784 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:32.384+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:32.781+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:32.781+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:32.781+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:33.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:32.782+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:33.384 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:33.076+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:33.384 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:33.076+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:33.384 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:33.077+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:33.384 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:33.078+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:33.773 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:33.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:33.773 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:33.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:33.773 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:33.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:33.773 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:33.382+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:34.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:33.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:34.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:33.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:34.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:33.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:34.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:33.771+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:34.430 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:34.129+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12470 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:34.431 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:34.088+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:34.431 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:34.088+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:34.431 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:34.088+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:34.431 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:34.089+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:34.767 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:34.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:34.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:34.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:34.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:34.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:34.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:34.428+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:34.765+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:34.765+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:34.765+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:34.766+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:35.465 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:35.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:35.465 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:35.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:35.465 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:35.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:35.465 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:35.070+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:35.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:35.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:35.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:35.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:35.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:35.462+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:35.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:35.463+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:35.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:35.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:35.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:35.778+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:36.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:36.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:36.488 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:36.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:36.488 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:36.079+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:36.488 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:36.079+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:36.485+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:36.485+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:36.485+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:36.486+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:36.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:36.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:36.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:36.825+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:37.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:37.043+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:37.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:37.043+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:37.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:37.043+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:37.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:37.043+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:37.785 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:37.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:37.785 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:37.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:37.785 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:37.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:37.785 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:37.511+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:38.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:38.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:38.026+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:38.027+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:37.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:37.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:37.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:38.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:37.783+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:38.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:38.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:38.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:38.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:38.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:38.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:38.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:38.522+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:39.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:38.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:39.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:38.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:39.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:38.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:39.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:38.778+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:39.525 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:39.130+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12518 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:39.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:39.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:39.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:39.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:39.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:39.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:39.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:39.068+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:39.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:39.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:39.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:39.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:39.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:39.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:39.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:39.524+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:39.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:39.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:39.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:39.802+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:40.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:40.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:40.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:40.037+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:40.491+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:40.491+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:40.491+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:40.492+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:40.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:40.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:40.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:40.820+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:41.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:41.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:41.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:41.065+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:41.800 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:41.508+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:41.800 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:41.508+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:41.800 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:41.508+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:41.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:41.510+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:41.797+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:41.797+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:41.797+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:41.798+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:42.041+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:42.041+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:42.041+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:42.041+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:42.762 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:42.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:42.762 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:42.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:42.762 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:42.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:42.762 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:42.533+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:42.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:42.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:42.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:42.760+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:43.551 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:43.063+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:43.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:43.063+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:43.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:43.063+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:43.552 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:43.063+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:43.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:43.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:43.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:43.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:43.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:43.549+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:43.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:43.550+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:43.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:43.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:43.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:43.802+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:44.559 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:44.131+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12568 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:44.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:44.063+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:44.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:44.063+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:44.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:44.063+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:44.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:44.063+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:44.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:44.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:44.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:44.557+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:44.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:44.772+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:44.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:44.772+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:44.772+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:44.773+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:45.433 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:45.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:45.434 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:45.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:45.434 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:45.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:45.434 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:45.064+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:45.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:45.520+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:45.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:45.520+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:45.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:45.520+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:45.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:45.521+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:45.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:45.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:45.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:45.738+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:46.553 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:46.068+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:46.554 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:46.068+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:46.554 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:46.068+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:46.554 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:46.070+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:46.751+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:46.751+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:46.751+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:46.752+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:46.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:46.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:46.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:46.551+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:47.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:47.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:47.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:47.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:47.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:47.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:47.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:47.108+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:47.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:47.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:47.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:47.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:47.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:47.515+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:47.783 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:47.516+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:47.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:47.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:47.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:47.781+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:48.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:48.060+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:48.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:48.060+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:48.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:48.060+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:48.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:48.061+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:48.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:48.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:48.556+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:48.557+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:48.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:48.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:48.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:48.736+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:49.132+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12619 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:49.024+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:49.024+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:49.024+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:49.025+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:49.728+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:49.728+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:49.728+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:49.729+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:49.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:49.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:49.585+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:49.586+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:50.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:50.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:50.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:50.062+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:50.561+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:50.561+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:50.561+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:50.562+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:50.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:50.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:50.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:50.981 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:50.762+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.013+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.013+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.013+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.014+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:51.988 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:51.584+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:51.988 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:51.584+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:51.988 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:51.584+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:51.988 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:51.585+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:51.989 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:51.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:51.989 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:51.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:51.989 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:51.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:51.989 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:51.723+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.985+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.985+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.985+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:51.986+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:52.705+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:52.705+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:52.705+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:52.706+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:52.557+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:52.557+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:52.557+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:52.558+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:52.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:52.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:52.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:52.982+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:53.732+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:53.732+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:53.732+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:53.733+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:53.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:53.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:53.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:53.519+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:54.133+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12672 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:53.959+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:53.959+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:53.959+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:53.959+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:54.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:54.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:54.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:54.539+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:54.783+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:54.783+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:54.783+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:54.784+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:55.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:55.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:55.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:55.008+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:55.834 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:55.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:55.834 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:55.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:55.834 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:55.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:55.834 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:55.567+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:55.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:55.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:55.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:55.832+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.021+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:56.877 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:56.606+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:56.877 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:56.606+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:56.878 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:56.606+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:56.878 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:56.607+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:56.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:56.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:56.875+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:56.876+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:56.992+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:58.044 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:57.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:58.044 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:57.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:58.044 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:57.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:58.044 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:57.651+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:58.044 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:57.865+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:58.044 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:57.865+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:58.044 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:57.865+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:58.045 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:57.866+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:58.041+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:58.041+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:58.041+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:58.042+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:46:58.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:58.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:58.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:58.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:58.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:58.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:46:58.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:58.898+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:58.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:58.692+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:58.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:58.692+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:58.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:58.692+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:46:58.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:58.693+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:46:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:46:59.134+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12724 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:46:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:59.022+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:59.022+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:59.022+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:46:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:46:59.023+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:00.017 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:59.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:00.017 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:59.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:00.017 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:59.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:00.017 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:46:59.666+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:00.018 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:59.895+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:00.018 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:59.895+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:00.018 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:59.895+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:00.018 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:46:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:46:59.896+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:00.014+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:00.014+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:00.014+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:00.015+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:01.054 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:00.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:01.054 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:00.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:01.054 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:00.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:01.054 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:00.638+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:01.055 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:00.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:01.055 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:00.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:01.055 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:00.856+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:01.055 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:00.857+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:01.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:01.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:01.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:01.064+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:01.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:01.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:01.852+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:01.853+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:01.666+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:01.666+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:01.666+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:02.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:01.666+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:02.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:02.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:02.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:02.093+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:02.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:02.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:02.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:02.895+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:02.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:02.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:02.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:02.650+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:03.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:03.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:03.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:03.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:03.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:03.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:03.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:03.067+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:03.854 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:03.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:03.854 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:03.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:03.854 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:03.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:03.855 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:03.649+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:04.138 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:03.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:04.139 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:03.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:04.139 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:03.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:04.139 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:03.853+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:04.139 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:04.097+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:04.139 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:04.097+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:04.139 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:04.097+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:04.139 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:04.098+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:04.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:04.136+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12777 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:04.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:04.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:04.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:04.650+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:04.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:04.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:04.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:05.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:04.872+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:05.081+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:05.081+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:05.081+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:05.082+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:05.681+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:05.681+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:05.681+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:05.682+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:05.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:05.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:05.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:05.884+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:06.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:06.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:06.123+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:06.125+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:06.932 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:06.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:06.932 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:06.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:06.932 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:06.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:06.932 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:06.675+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:07.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:07.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:07.170+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:07.171+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:06.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:06.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:07.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:06.929+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:07.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:06.930+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:07.942 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:07.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:07.942 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:07.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:07.942 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:07.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:07.942 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:07.688+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:08.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:07.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:08.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:07.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:08.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:07.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:08.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:07.940+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:08.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:08.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:08.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:08.215+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:08.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:08.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:08.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:08.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:08.909+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:08.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:08.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:08.694+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:08.694+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:09.137+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12829 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:09.206+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:09.206+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:09.206+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:09.207+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:09.937 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:09.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:09.937 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:09.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:09.937 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:09.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:09.937 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:09.676+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:10.208 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:09.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:10.208 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:09.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:10.208 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:09.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:10.208 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:09.935+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:10.205+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:10.205+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:10.205+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:10.206+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:10.938 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:10.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:10.939 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:10.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:10.939 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:10.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:10.939 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:10.634+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:11.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:10.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:11.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:10.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:11.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:10.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:11.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:10.937+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:11.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:11.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:11.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:11.232+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:11.931 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:11.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:11.931 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:11.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:11.931 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:11.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:11.931 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:11.635+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:12.187 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:11.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:12.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:11.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:12.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:11.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:12.188 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:11.929+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:12.186+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:12.186+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:12.186+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:12.186+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:12.928 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:12.588+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:12.928 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:12.588+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:12.928 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:12.588+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:12.928 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:12.588+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:13.216 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:12.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:13.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:12.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:13.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:12.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:13.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:12.926+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:13.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:13.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:13.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:13.214+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:13.936 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:13.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:13.936 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:13.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:13.936 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:13.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:13.936 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:13.615+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:14.258 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:14.137+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12882 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:13.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:13.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:13.932+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:14.258 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:13.933+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:14.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:14.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:14.255+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:14.255+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:14.912 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:14.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:14.912 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:14.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:14.912 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:14.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:14.912 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:14.649+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:15.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:14.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:15.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:14.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:15.244 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:14.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:15.245 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:14.910+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:15.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:15.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:15.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:15.242+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:15.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:15.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:15.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:15.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:15.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:15.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:15.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:15.668+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:16.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:15.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:16.280 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:15.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:16.280 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:15.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:16.280 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:15.941+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:16.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:16.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:16.277+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:16.278+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:16.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:16.628+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:16.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:16.628+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:16.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:16.628+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:16.981 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:16.629+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:17.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:16.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:17.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:16.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:17.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:16.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:17.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:16.979+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:17.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:17.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:17.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:17.267+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:17.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:17.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:17.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:17.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:17.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:17.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:17.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:17.651+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:18.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:17.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:18.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:17.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:18.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:17.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:18.241 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:17.955+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:18.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:18.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:18.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:18.239+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:18.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:18.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:18.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:18.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:18.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:18.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:18.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:18.649+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:19.226 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:19.139+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12934 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:19.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:19.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:19.226 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:19.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:19.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:19.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:19.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:19.205+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:19.227 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:18.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:19.227 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:18.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:19.227 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:18.979+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:19.227 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:18.980+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:19.951 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:19.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:19.952 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:19.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:19.952 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:19.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:19.952 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:19.680+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:20.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:19.949+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:20.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:19.949+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:20.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:19.949+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:20.226 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:19.950+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:20.222+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:20.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:20.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:20.223+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:20.951 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:20.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:20.951 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:20.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:20.951 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:20.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:20.951 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:20.645+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:21.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:21.192+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:21.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:21.192+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:21.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:21.192+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:21.210 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:21.193+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:21.210 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:20.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:21.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:20.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:21.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:20.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:21.211 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:20.948+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:21.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:21.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:21.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:21.937+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:21.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:21.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:21.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:21.688+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:22.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:22.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:22.231+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:22.232+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:22.904+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:22.904+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:22.904+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:22.904+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:22.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:22.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:23.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:22.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:23.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:22.656+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:23.260+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:23.260+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:23.260+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:23.261+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:23.952 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:23.646+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:23.952 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:23.646+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:23.952 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:23.646+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:23.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:23.646+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:24.214 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:24.141+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 12987 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:24.215 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:23.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:24.215 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:23.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:24.215 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:23.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:24.215 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:23.950+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:24.212+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:24.212+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:24.212+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:24.213+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:24.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:24.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:24.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:24.663+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:24.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:25.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:24.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:25.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:24.907+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:25.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:24.908+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:25.260+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:25.260+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:25.260+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:25.261+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:25.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:25.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:25.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:25.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:25.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:25.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:25.925 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:25.638+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:26.247 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:25.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:26.247 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:25.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:26.248 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:25.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:26.248 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:25.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:26.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:26.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:26.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:26.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:26.306+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:26.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:26.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:26.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:26.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:26.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:26.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:26.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:26.674+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:26.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:26.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:26.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:26.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:27.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:27.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:27.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:27.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:27.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:27.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:27.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:27.329+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:27.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:27.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:27.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:27.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:27.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:27.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:27.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:27.652+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:27.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:27.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:27.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:27.921+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:28.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:28.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:28.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:28.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:28.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:28.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:28.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:28.360+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:28.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:28.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:28.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:28.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:28.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:28.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:28.898+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:28.899+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:29.142+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13039 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:29.701 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:29.321+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:29.701 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:29.321+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:29.701 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:29.321+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:29.701 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:29.322+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:29.698+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:29.698+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:29.698+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:29.699+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:29.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:29.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:29.901+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:30.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:29.902+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:30.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:30.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:30.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:30.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:30.727+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:30.727+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:30.727+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:30.728+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:30.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:31.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:30.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:31.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:30.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:31.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:30.940+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:31.270+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:31.270+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:31.270+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:31.271+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:31.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:31.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:31.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:32.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:31.934+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:31.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:31.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:31.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:32.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:31.716+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:32.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:32.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:32.249+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:32.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:32.250+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:32.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:32.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:32.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:32.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:32.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:32.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:32.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:32.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:33.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:32.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:33.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:32.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:33.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:32.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:33.285 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:32.982+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:33.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:33.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:33.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:33.283+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:34.032 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:33.731+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:34.032 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:33.731+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:34.032 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:33.731+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:34.032 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:33.732+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:34.143+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13091 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:34.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:34.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:34.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:34.030+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:34.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:34.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:34.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:34.260+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:34.709+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:34.709+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:34.709+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:34.710+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:35.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:35.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:35.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:35.073+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:35.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:35.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:35.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:35.257+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:35.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:35.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:35.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:35.721+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:36.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:36.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:36.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:36.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:36.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:36.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:36.237+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:36.238+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:36.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:36.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:36.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:36.734+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:37.107+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:37.107+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:37.107+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:37.107+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:37.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:37.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:37.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:37.193+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:37.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:37.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:37.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:37.688+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:38.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:38.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:38.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:38.114+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:38.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:38.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:38.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:38.191+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:39.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:38.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:39.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:38.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:39.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:38.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:39.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:38.735+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:39.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:39.144+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13143 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:39.080+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:39.080+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:39.080+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:39.080+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:39.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:39.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:39.236+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:39.237+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:40.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:39.757+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:40.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:39.757+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:40.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:39.757+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:40.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:39.757+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:40.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:40.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:40.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:40.245+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:40.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:40.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:40.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:40.041+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:40.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:40.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:40.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:40.717+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:41.362 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:41.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:41.362 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:41.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:41.363 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:41.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:41.363 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:41.244+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:41.363 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:41.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:41.363 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:41.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:41.363 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:41.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:41.363 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:41.062+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:41.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:41.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:41.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:41.750+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:42.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:42.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:42.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:42.258+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:42.102+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:42.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:42.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:42.103+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:42.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:42.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:42.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:42.762+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:43.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:43.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:43.275+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:43.276+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:43.073+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:43.073+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:43.073+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:43.074+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:43.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:43.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:43.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:43.800+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:44.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:44.146+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13196 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:44.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:44.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:44.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:44.098+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:44.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:44.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:44.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:44.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:44.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:44.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:44.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:44.305+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:44.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:44.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:44.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:44.763+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:45.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:45.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:45.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:45.330+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:45.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:45.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:45.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:45.087+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:45.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:45.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:45.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:45.765+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:46.401 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:46.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:46.401 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:46.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:46.401 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:46.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:46.401 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:46.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:46.401 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:46.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:46.401 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:46.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:46.402 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:46.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:46.402 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:46.316+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:47.161 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:46.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:47.161 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:46.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:47.162 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:46.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:47.162 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:46.815+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:47.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:47.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:47.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:47.160+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:47.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:47.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:47.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:47.341+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:48.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:47.811+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:48.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:47.811+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:48.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:47.811+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:48.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:47.812+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:48.148 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:48.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:48.148 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:48.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:48.148 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:48.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:48.146+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:48.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:48.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:48.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:48.367+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:48.807+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:48.807+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:48.807+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:48.808+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:49.147+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13248 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:49.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:49.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:49.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:49.146+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:49.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:49.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:49.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:49.366+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:49.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:49.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:49.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:49.769+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:50.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:50.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:50.116+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:50.117+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:50.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:50.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:50.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:50.321+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:50.730+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:51.439 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:51.080+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:51.439 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:51.080+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:51.439 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:51.080+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:51.441 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:51.081+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:51.441 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:51.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:51.441 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:51.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:51.441 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:51.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:51.441 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:51.273+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:51.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:51.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:51.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:51.766+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:52.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:52.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:52.239+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:52.239+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:52.126+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:52.126+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:52.126+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:52.127+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:52.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:52.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:52.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:52.722+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:53.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:53.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:53.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:53.195+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:53.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:53.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:53.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:53.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:53.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:53.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:53.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:53.677+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:54.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:54.148+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13301 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:54.063+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:54.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:54.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:54.064+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:54.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:54.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:54.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:54.214+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:54.685+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:54.685+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:54.685+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:54.686+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:55.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:55.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:55.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:55.106+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:55.172+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:55.172+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:55.172+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:55.173+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:55.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:55.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:55.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:55.643+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:56.476 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:56.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:56.476 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:56.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:56.477 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:56.191+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:56.477 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:56.192+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:56.477 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:56.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:56.477 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:56.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:56.477 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:56.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:56.477 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:56.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:56.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:56.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:56.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:56.595+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:57.226+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:57.226+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:57.226+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:57.227+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:57.089+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:57.089+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:57.089+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:57.089+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:57.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:57.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:57.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:57.581+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:58.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:58.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:58.112+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:58.112+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:58.201+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:58.201+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:58.201+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:58.201+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:47:58.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:58.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:58.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:58.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:58.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:58.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:47:58.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:58.583+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:47:59.149+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13350 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:47:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:59.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:59.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:59.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:47:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:47:59.104+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:47:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:59.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:59.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:59.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:47:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:47:59.204+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:59.572+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:59.572+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:59.572+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:47:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:47:59.572+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:00.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:00.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:00.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:00.059+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:00.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:00.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:00.243+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:00.243+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:00.600+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:00.601+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:00.601+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:00.602+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:01.511 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:01.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:01.512 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:01.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:01.512 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:01.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:01.512 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:01.067+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:01.512 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:01.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:01.512 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:01.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:01.512 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:01.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:01.512 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:01.249+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:01.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:01.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:01.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:01.647+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:02.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:02.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:02.078+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:02.079+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:02.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:02.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:02.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:02.223+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:03.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:02.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:03.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:02.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:03.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:02.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:03.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:02.612+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:03.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:03.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:03.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:03.036+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:03.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:03.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:03.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:03.258+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:03.605+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:03.605+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:03.605+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:03.605+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:04.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:04.151+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13400 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:04.220+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:04.220+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:04.220+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:04.221+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:04.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:04.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:04.059+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:04.060+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:05.042 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:04.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:05.042 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:04.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:05.042 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:04.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:05.042 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:04.593+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:05.305 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:05.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:05.306 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:05.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:05.306 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:05.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:05.306 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:05.040+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:05.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:05.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:05.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:05.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:05.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:05.238+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:05.306 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:05.239+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:06.038 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:05.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:06.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:05.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:06.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:05.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:06.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:05.615+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:06.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:06.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:06.036+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:06.037+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:06.227+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:06.227+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:06.227+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:06.229+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:06.625+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:06.625+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:06.625+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:06.626+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:07.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:07.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:07.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:07.031+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:07.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:07.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:07.213+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:07.214+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:07.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:07.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:07.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:07.658+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:08.378 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:08.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:08.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:08.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:08.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:08.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:08.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:08.073+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:08.379 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:08.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:08.379 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:08.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:08.379 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:08.229+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:08.379 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:08.230+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:08.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:08.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:08.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:08.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:08.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:08.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:08.625+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:09.152+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13446 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:09.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:09.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:09.203+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:09.204+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:09.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:09.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:09.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:09.110+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:09.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:09.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:09.604+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:09.605+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:10.073+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:10.073+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:10.073+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:10.074+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:10.174+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:10.174+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:10.174+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:10.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:10.175+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:10.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:10.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:10.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:10.612+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:11.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:11.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:11.066+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:11.067+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:11.185+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:11.185+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:11.185+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:11.185+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:11.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:11.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:11.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:11.658+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:12.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:12.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:12.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:12.070+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:12.207+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:12.207+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:12.207+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:12.208+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:12.628+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:12.628+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:12.628+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:12.628+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:13.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:13.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:13.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:13.077+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:13.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:13.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:13.257+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:13.257+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:13.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:13.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:13.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:13.622+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:14.522 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:14.153+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13489 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:14.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:14.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:14.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:14.306+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:14.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:14.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:14.081+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:14.523 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:14.082+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:14.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:14.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:14.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:14.656+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:15.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:15.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:15.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:15.118+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:15.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:15.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:15.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:15.272+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:15.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:15.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:15.701+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:15.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:16.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:16.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:16.225+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:16.225+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:16.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:16.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:16.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:16.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:16.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:16.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:16.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:16.681+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:17.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:17.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:17.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:17.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:17.180+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:17.180+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:17.180+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:17.181+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:17.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:17.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:17.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:17.638+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:18.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:18.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:18.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:18.137+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:18.167+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:18.167+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:18.167+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:18.167+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:18.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:18.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:18.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:18.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:18.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:18.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:18.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:18.675+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:19.154+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13532 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:19.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:19.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:19.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:19.093+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:19.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:19.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:19.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:19.136+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:19.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:19.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:19.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:19.638+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:20.124+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:20.124+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:20.124+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:20.124+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:20.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:20.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:20.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:20.137+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:20.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:20.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:20.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:20.595+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:21.075+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:21.075+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:21.075+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:21.076+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:21.130+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:21.130+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:21.130+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:21.131+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:22.033 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:21.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:22.033 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:21.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:22.033 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:21.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:22.033 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:21.575+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:22.134+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:22.134+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:22.134+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:22.135+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:22.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:22.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:22.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:22.031+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:22.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:22.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:22.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:22.548+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:23.116+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:23.116+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:23.116+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:23.116+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:23.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:23.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:23.031+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:23.032+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:24.008 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:23.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:24.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:23.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:24.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:23.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:24.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:23.564+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:24.156+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13578 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:24.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:24.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:24.107+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:24.108+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.006+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:24.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:24.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:24.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:24.518+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:24.969+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:25.110+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:25.110+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:25.110+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:25.111+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:25.520+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:25.520+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:25.520+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:25.521+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:26.083+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:26.083+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:26.083+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:26.084+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:25.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:25.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:25.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:26.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:25.999+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:26.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:26.473+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:26.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:26.473+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:26.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:26.473+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:26.473+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:27.105+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:27.105+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:27.105+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:27.106+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:27.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:27.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:27.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:27.024+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:27.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:27.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:27.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:27.425+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:28.464 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:28.142+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:28.464 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:28.142+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:28.464 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:28.142+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:28.464 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:28.143+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:28.465 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:28.070+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:28.465 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:28.070+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:28.465 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:28.070+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:28.465 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:28.070+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:28.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:28.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:28.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:28.462+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:29.157+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13625 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:29.157+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:29.157+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:29.157+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:29.159+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:29.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:29.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:29.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:29.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:29.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:29.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:29.114+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:29.478+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:29.478+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:29.478+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:29.480+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:30.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:30.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:30.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:30.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:30.085+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:30.528 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:30.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:30.528 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:30.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:30.528 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:30.194+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:30.529 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:30.195+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:30.525+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:30.525+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:30.525+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:30.526+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:31.501 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:31.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:31.502 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:31.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:31.502 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:31.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:31.502 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:31.108+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:31.502 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:31.156+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:31.502 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:31.156+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:31.502 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:31.156+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:31.502 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:31.156+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:31.499+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:31.499+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:31.499+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:31.499+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:32.442 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:32.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:32.443 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:32.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:32.443 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:32.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:32.443 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:32.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:32.443 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:32.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:32.443 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:32.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:32.443 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:32.136+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:32.443 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:32.137+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:32.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:32.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:32.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:32.517+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:33.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:33.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:33.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:33.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:33.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:33.153+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:33.517 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:33.154+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:33.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:33.121+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:33.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:33.121+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:33.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:33.121+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:33.518 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:33.122+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:33.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:33.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:33.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:33.516+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:34.504 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:34.158+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13677 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:34.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:34.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:34.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:34.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:34.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:34.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:34.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:34.157+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:34.505 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:34.104+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:34.505 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:34.104+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:34.505 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:34.104+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:34.506 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:34.105+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:34.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:34.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:34.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:34.503+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:35.485 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:35.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:35.485 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:35.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:35.485 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:35.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:35.485 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:35.115+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:35.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:35.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:35.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:35.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:35.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:35.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:35.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:35.119+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:35.483+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:35.483+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:35.483+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:35.483+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:36.097+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:36.097+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:36.097+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:36.097+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:36.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:36.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:36.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:36.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:36.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:36.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:36.527+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:36.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:36.527+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:36.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:36.527+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:36.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:36.528+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:37.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:37.066+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:37.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:37.066+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:37.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:37.066+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:37.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:37.067+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:37.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:37.149+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:37.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:37.149+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:37.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:37.149+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:37.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:37.150+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:37.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:37.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:37.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:37.485+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:38.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:38.099+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:38.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:38.099+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:38.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:38.099+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:38.487 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:38.100+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:38.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:38.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:38.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:38.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:38.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:38.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:38.488 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:38.120+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:38.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:38.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:38.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:38.485+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:39.159+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13730 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:39.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:39.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:39.067+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:39.067+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:39.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:39.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:39.103+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:39.103+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:39.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:39.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:39.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:39.498+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:40.505 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:40.068+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:40.505 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:40.068+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:40.505 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:40.068+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:40.505 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:40.069+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:40.505 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:40.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:40.506 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:40.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:40.506 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:40.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:40.506 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:40.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:40.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:40.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:40.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:40.503+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:41.058+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:41.058+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:41.058+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:41.058+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:41.085+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:41.085+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:41.085+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:41.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:41.086+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:41.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:41.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:41.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:41.490+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:42.085 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:42.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:42.085 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:42.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:42.085 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:42.055+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:42.085 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:42.056+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:42.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:42.082+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:42.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:42.082+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:42.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:42.082+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:42.479 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:42.083+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:42.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:42.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:42.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:42.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:42.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:42.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:42.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:42.477+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:43.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:43.096+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:43.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:43.096+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:43.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:43.096+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:43.454 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:43.097+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:43.455 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:43.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:43.455 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:43.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:43.455 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:43.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:43.455 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:43.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:43.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:43.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:43.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:43.452+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:44.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:44.161+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13782 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:44.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:44.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:44.029+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:44.030+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:44.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:44.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:44.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:44.114+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:44.420+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:44.420+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:44.420+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:44.421+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.001+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.002+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:45.080+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:45.080+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:45.080+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:45.081+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:45.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:45.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:45.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:45.397+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:45.976+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:46.054+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:46.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:46.054+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:46.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:46.054+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:46.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:46.055+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:46.397+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:46.397+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:46.397+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:46.398+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:46.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:46.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:46.971+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:46.972+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:47.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:47.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:47.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:47.038+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:47.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:47.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:47.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:47.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:47.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:47.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:47.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:47.427+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:48.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:48.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:48.014+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:48.014+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:48.055+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:48.055+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:48.055+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:48.056+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:48.468+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:48.468+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:48.468+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:48.469+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:49.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:49.162+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13835 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:49.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:49.051+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:49.051+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:49.051+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:49.052+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:48.999+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:48.999+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:48.999+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:49.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:49.000+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:49.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:49.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:49.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:49.517+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:49.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:49.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:49.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:49.983+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:50.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:50.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:50.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:50.073+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:50.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:50.074+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:50.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:50.561+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:50.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:50.561+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:50.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:50.561+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:50.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:50.562+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:50.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:50.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:50.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:50.953+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:51.032+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:51.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:51.032+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:51.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:51.032+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:51.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:51.032+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:51.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:51.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:51.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:51.575+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:51.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:51.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:51.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:51.910+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:52.037+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:52.037+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:52.037+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:52.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:52.038+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:52.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:52.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:52.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:52.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:52.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:52.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:52.957 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:52.563+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:53.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:53.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:53.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:53.070+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:52.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:52.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:52.954+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:53.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:52.955+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:53.552+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:53.552+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:53.552+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:53.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:53.553+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:54.163+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13887 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:53.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:53.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:53.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:53.981+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:54.023+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:54.023+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:54.023+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:54.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:54.024+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:54.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:54.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:54.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:54.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:54.535+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:55.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:55.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:55.064+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:55.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:55.065+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:55.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:55.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:55.027+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:55.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:55.028+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:55.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:55.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:55.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:55.546+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.031+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:56.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:56.048+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:56.049+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:56.049+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:56.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:56.049+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:56.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:56.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:56.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:56.538+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.985+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.985+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.985+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:56.987+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:57.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:57.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:57.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:57.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:57.075+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:57.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:57.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:57.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:57.549+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:58.065+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:58.065+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:58.065+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:58.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:58.066+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:57.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:57.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:58.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:57.952+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:58.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:57.953+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:58.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:58.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:58.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:58.524+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:48:59.164+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13940 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:48:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:58.965+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:58.965+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:58.965+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:48:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:58.966+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:48:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:59.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:59.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:59.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:48:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:48:59.070+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:48:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:59.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:59.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:59.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:48:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:48:59.519+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:00.086+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:00.086+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:00.086+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:00.087+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:59.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:59.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:59.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:00.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:48:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:48:59.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:00.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:00.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:00.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:00.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:00.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:00.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:00.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:00.566+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:00.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:00.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:00.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:00.942+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:01.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:01.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:01.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:01.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:01.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:01.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:01.062+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:01.551+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:01.551+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:01.551+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:01.552+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:02.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.014+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:02.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.014+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.014+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.015+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:01.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:01.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:01.956+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:02.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:01.957+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:02.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:02.558+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:02.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:02.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:02.945 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:02.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:02.945 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:02.559+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:02.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:02.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:02.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:02.942+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:02.997+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:03.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:03.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:03.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:03.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:03.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:03.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:03.956 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:03.572+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:04.166+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 13992 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.015+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.015+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.015+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.015+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:03.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:03.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:03.953+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:03.954+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:04.927 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:04.563+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:04.928 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:04.563+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:04.928 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:04.563+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:04.928 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:04.565+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:04.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:04.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:04.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:04.925+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.984+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:05.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.984+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:05.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.984+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:05.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:04.984+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:05.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:05.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:05.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:05.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:05.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:05.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:05.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:05.592+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:05.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:05.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:05.941+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:05.941+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:06.029+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:06.029+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:06.029+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:06.030+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:06.916 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:06.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:06.916 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:06.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:06.916 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:06.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:06.916 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:06.615+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:07.066+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:07.066+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:07.066+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:07.067+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:06.913+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:06.913+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:06.913+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:07.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:06.914+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:07.927 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:07.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:07.929 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:07.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:07.929 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:07.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:07.929 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:07.568+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:08.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:08.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:08.038+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:08.039+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:07.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:07.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:07.924+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:07.925+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:08.890 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:08.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:08.890 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:08.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:08.890 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:08.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:08.891 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:08.597+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:09.170 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:08.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:09.170 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:08.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:09.170 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:08.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:09.170 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:08.889+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:09.170 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:08.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:09.170 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:08.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:09.170 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:08.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:09.171 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:09.000+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:09.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:09.167+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14045 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:09.818 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:09.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:09.818 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:09.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:09.818 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:09.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:09.818 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:09.564+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:09.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:09.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:09.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:09.871+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:10.019+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:10.019+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:10.019+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:10.019+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:10.914 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:10.602+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:10.914 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:10.602+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:10.915 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:10.602+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:10.915 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:10.603+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.028+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.028+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.028+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.029+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:10.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:10.911+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:10.912+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:10.912+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:11.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:11.555+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:11.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:11.555+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:11.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:11.555+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:11.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:11.556+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:11.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:11.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:11.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:11.903+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:12.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:11.998+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:12.531+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:12.531+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:12.531+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:12.532+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:12.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:12.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:12.922+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:13.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:12.923+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:13.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:13.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:13.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:13.001+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:13.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:13.001+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:13.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:13.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:13.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:13.514+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:14.171 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:13.876+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:14.171 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:13.876+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:14.171 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:13.876+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:14.171 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:13.877+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:14.172 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:14.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:14.172 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:14.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:14.172 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:14.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:14.172 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:14.004+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:14.538 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:14.168+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14097 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:14.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:14.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:14.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:14.536+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:14.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:14.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:14.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:14.852+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:15.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:15.050+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:15.050+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:15.050+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:15.051+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:15.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:15.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:15.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:15.511+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:15.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:15.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:15.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:15.897+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:16.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:16.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:16.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:16.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:16.003+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:16.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:16.004+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:16.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:16.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:16.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:16.465+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:16.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:16.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:16.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:16.910+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.025+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.025+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.025+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.026+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:17.429+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:17.429+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:17.429+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:17.430+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.976+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.976+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.976+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:17.977+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:17.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:17.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:17.934+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:18.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:17.935+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:18.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:18.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:18.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:18.425+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:19.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:19.170+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14150 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:19.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.008+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.008+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.008+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.009+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:19.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:18.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:19.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:18.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:19.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:18.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:19.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:18.963+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:19.421+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:19.421+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:19.421+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:19.421+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:19.965+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:19.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:19.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:19.975+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:19.976+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:20.445+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:20.445+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:20.445+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:20.446+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:20.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:20.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:20.947+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:20.948+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:20.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:20.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:20.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:20.996+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:21.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:21.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:21.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:21.425+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:21.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:21.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:21.983+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:21.984+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:21.963+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:21.963+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:21.963+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:21.964+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:22.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:22.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:22.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:22.409+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:22.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:22.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:22.996+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:22.997+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:22.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:22.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:22.958+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:22.958+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:23.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:23.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:23.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:23.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:24.171+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14202 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:23.944+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:23.944+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:23.944+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:23.944+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:24.021+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:24.021+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:24.021+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:24.022+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:24.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:24.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:24.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:24.319+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:24.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:24.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:24.988+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:24.988+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:25.034+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:25.034+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:25.034+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:25.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:25.035+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:25.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:25.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:25.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:25.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:26.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:26.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:26.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:26.011+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.011+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.011+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.011+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:26.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.012+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:26.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:26.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:26.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:26.325+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.013+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.978+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.978+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.978+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:26.979+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:27.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:27.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:27.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:27.371+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:28.250 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:28.250 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:28.251 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.986+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:28.251 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:27.986+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:28.251 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:27.951+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:28.251 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:27.951+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:28.251 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:27.951+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:28.251 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:27.952+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:28.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:28.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:28.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:28.357+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:29.172+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14255 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.030+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.030+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:28.956+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:28.956+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:28.956+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:28.956+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:29.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:29.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:29.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:29.363+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.991+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.991+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.991+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:29.992+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:29.915+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:29.915+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:29.915+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:29.916+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:30.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:30.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:30.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:30.385+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:30.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:30.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:30.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:30.908+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:31.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.014+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:31.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:31.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:31.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:31.389+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:31.923+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:31.923+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:31.923+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:31.924+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.967+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.967+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.967+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:32.212 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:31.968+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:32.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:32.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:32.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:32.382+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:32.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:32.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:32.955+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:32.955+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:33.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:33.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:33.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:33.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:33.004+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:33.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:33.005+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:33.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:33.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:33.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:33.415+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:34.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:34.173+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14307 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:33.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:33.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:33.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:33.959+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:34.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:34.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:34.041+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:34.041+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:34.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:34.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:34.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:34.401+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:35.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:35.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:35.072+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:35.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:34.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:34.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:34.999+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:35.000+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:35.403+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:35.403+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:35.403+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:35.404+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:36.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:36.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:36.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:36.068+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.007+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:36.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.008+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:36.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:36.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:36.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:36.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:36.377+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:37.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.979+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:37.249 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.979+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:37.249 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.979+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:37.249 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:36.979+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:37.249 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:37.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:37.249 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:37.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:37.249 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:37.098+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:37.249 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:37.099+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:37.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:37.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:37.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:37.403+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:38.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:38.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:38.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:38.123+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:37.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:37.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:38.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:37.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:38.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:37.948+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:38.366+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:38.366+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:38.366+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:38.367+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:39.167 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:38.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:39.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:38.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:39.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:38.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:39.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:38.923+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:39.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:39.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:39.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:39.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:39.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:39.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:39.168 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:39.163+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:39.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:39.174+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14360 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:39.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:39.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:39.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:39.340+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:40.183 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:39.920+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:40.183 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:39.920+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:40.183 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:39.920+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:40.183 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:39.920+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:40.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:40.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:40.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:40.180+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:40.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:40.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:40.318+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:40.319+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:41.155+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:41.155+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:41.155+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:41.156+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:40.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:40.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:40.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:40.946+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:41.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:41.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:41.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:41.312+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:42.191 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:41.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:42.192 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:41.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:42.192 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:41.901+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:42.192 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:41.901+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:42.290+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:42.290+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:42.290+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:42.290+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:42.189+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:42.189+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:42.189+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:42.190+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:43.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:42.902+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:43.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:42.902+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:43.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:42.902+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:43.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:42.903+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:43.320+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:43.320+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:43.320+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:43.321+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:43.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:43.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:43.179+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:43.180+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:44.165 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:43.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:44.166 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:43.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:44.166 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:43.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:44.166 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:43.889+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:44.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:44.176+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14412 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:44.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:44.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:44.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:44.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:44.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:44.334+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:44.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:44.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:44.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:44.164+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:45.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:44.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:45.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:44.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:45.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:44.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:45.186 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:44.893+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:45.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:45.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:45.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:45.328+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:45.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:45.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:45.183+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:45.184+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:46.191 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:45.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:46.191 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:45.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:46.191 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:45.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:46.191 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:45.912+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:46.188+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:46.188+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:46.188+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:46.189+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:46.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:46.364+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:46.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:46.364+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:46.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:46.364+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:46.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:46.365+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:47.228 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:46.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:47.228 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:46.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:47.228 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:46.883+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:47.228 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:46.883+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:47.225+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:47.225+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:47.225+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:47.226+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:47.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:47.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:47.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:47.380+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:48.257 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:47.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:48.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:47.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:48.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:47.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:48.258 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:47.908+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:48.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:48.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:48.255+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:48.255+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:48.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:48.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:48.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:48.364+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:49.180 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:48.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:49.180 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:48.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:49.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:48.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:49.181 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:48.895+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:49.178+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14465 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:49.404+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:49.404+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:49.404+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:49.405+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:49.280+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:49.280+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:49.280+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:49.281+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:49.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:49.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:49.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:49.870+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:50.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:50.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:50.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:50.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:50.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:50.300+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:50.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:50.301+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:50.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:50.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:50.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:50.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:50.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:50.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:50.779 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:50.423+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:51.262 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:50.840+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:51.262 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:50.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:51.262 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:50.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:51.262 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:50.841+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:51.259+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:51.259+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:51.259+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:51.260+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:51.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:51.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:51.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:51.419+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:52.219 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:51.865+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:52.220 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:51.865+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:52.220 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:51.865+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:52.220 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:51.866+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:52.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:52.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:52.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:52.383+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:52.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:52.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:52.217+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:52.218+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:52.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:52.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:52.836+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:52.837+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:53.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:53.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:53.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:53.426+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:53.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:53.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:53.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:53.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:53.211+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:54.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:53.863+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:54.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:53.863+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:54.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:53.863+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:54.182 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:53.864+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:54.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:54.179+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14517 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:54.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:54.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:54.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:54.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:54.234+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:54.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:54.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:54.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:54.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:54.412+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:55.216 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:54.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:55.216 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:54.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:55.216 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:54.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:55.217 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:54.876+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:55.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:55.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:55.412+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:55.412+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:55.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:55.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:55.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:55.214+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:56.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:55.893+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:56.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:55.893+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:56.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:55.893+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:56.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:55.894+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:56.430 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:56.166+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:56.430 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:56.166+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:56.430 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:56.166+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:56.430 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:56.167+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:56.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:56.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:56.427+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:56.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:56.428+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:57.205 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:56.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:57.206 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:56.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:57.206 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:56.921+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:57.206 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:56.921+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:57.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:57.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:57.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:57.379+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:57.203+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:57.203+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:57.203+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:57.204+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:58.165 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:57.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:58.165 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:57.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:58.165 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:57.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:58.165 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:57.890+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:58.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:58.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:58.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:58.411+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:58.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:58.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:58.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:58.163+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:59.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:49:59.180+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14570 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:49:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:58.933+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:58.933+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:58.933+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:49:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:58.934+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:49:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:59.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:59.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:59.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:59.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:59.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:49:59.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:49:59.140+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:49:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:59.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:59.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:59.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:49:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:49:59.380+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:59.963+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:59.963+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:59.963+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:49:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:49:59.963+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:00.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:00.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:00.122+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:00.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:00.123+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:00.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:00.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:00.371+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:00.371+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:01.315 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:00.968+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:01.315 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:00.968+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:01.315 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:00.968+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:01.315 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:00.969+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:01.315 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:01.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:01.315 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:01.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:01.316 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:01.106+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:01.316 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:01.106+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:01.374+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:01.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:01.374+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:01.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:01.374+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:01.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:01.375+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:02.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:02.093+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:02.093+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:02.093+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:02.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:01.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:01.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:01.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:02.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:01.989+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:02.419+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:02.419+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:02.419+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:02.420+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:03.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:03.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:03.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:03.062+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:02.982+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:02.982+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:03.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:02.982+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:03.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:02.983+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:03.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:03.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:03.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:03.429+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:04.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:04.182+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14622 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.021+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.021+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.021+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.022+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:04.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.028+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:04.440+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:04.440+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:04.440+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:04.440+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.992+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:04.993+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.996+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.996+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.996+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:04.997+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:05.405+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:05.405+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:05.405+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:05.405+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:06.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:06.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:06.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:06.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:06.032+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:06.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:06.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:06.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:06.047+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:06.650 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:06.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:06.650 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:06.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:06.650 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:06.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:06.650 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:06.387+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:07.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:07.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:07.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:07.018+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:07.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:07.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:07.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:07.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:07.036+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:07.686 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:07.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:07.686 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:07.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:07.686 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:07.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:07.686 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:07.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:08.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:08.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:08.034+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:08.035+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:08.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:08.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:08.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:08.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:08.021+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:08.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:08.022+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:08.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:08.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:08.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:08.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:08.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:08.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:08.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:08.364+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:09.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:09.183+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14675 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:09.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:09.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:09.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:09.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:09.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:09.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:09.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:09.041+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:09.037+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:09.037+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:09.037+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:09.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:09.038+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:09.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:09.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:09.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:09.381+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:10.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:10.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:10.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:10.077+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:10.049+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:10.049+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:10.049+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:10.050+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:10.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:10.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:10.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:10.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:10.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:10.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:10.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:10.377+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:10.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-12T03:50:10.817+0000 7fbc541ba700 0 monclient(hunting): authenticate timed out after 300 2024-03-12T03:50:10.822 INFO:teuthology.orchestra.run.smithi118.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-12T03:50:11.073 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:11.006+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:11.073 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:11.006+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:11.073 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:11.006+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:11.073 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:11.006+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:11.401 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:11.083+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:11.402 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:11.083+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:11.402 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:11.083+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:11.402 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:11.084+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:50:11.564 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 1423, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 97, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-12T03:50:11.790 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c1161e9a464c4e0d82845819cd66a215 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 1423, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 97, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-12T03:50:11.794 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-12T03:50:11.839 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-12T03:50:11.840 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:50:11.840 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T03:50:11.844 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:11.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:11.845 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:11.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:11.845 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:11.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:11.845 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:11.400+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:11.861 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:50:11.861 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T03:50:11.918 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T03:50:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:12.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:12.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:12.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:12.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:12.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:12.065+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:12.046+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:12.046+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:12.046+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:12.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:12.047+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:12.654 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:12.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:12.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:12.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:12.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:12.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:12.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:12.409+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:13.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:13.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:13.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:13.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:13.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:13.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:13.386 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:13.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:13.386 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:13.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:13.386 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:13.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:13.386 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:13.069+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:13.386 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:13.069+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:13.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:13.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:13.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:13.383+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:14.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:14.184+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14727 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:14.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:14.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:14.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:14.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:14.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:14.156+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.019+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.019+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.019+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:14.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.020+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:14.588 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:14.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:14.588 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:14.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:14.589 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:14.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:14.589 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:14.391+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:15.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:15.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:15.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:15.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:15.191+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:15.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:14.973+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:15.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:15.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:15.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:15.400+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:16.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:15.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:16.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:15.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:16.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:15.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:16.227 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:15.933+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:16.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:16.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:16.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:16.358+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:16.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:16.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:16.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:16.225+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:16.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:16.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:16.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:17.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:16.981+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:17.226+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:17.226+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:17.226+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:17.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:17.227+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:17.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:17.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:17.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:17.387+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:18.231 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:17.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:18.231 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:17.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:18.231 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:17.938+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:18.232 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:17.938+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:18.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:18.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:18.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:18.359+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:18.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:18.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:18.228+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:18.229+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:19.188 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:18.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:19.189 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:18.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:19.189 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:18.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:19.189 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:18.935+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:19.186+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14780 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:19.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:19.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:19.381+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:19.382+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:19.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:19.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:19.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:19.258+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:20.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:19.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:20.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:19.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:20.246 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:19.981+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:20.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:19.982+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:20.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:20.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:20.243+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:20.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:20.244+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:20.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:20.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:20.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:20.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:20.411+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:21.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:20.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:21.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:20.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:21.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:20.972+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:21.239 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:20.973+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:21.236+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:21.236+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:21.236+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:21.237+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:21.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:21.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:21.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:21.397+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:22.264 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:21.974+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:22.264 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:21.974+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:22.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:21.974+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:22.265 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:21.974+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:22.523 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:22.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:22.524 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:22.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:22.524 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:22.261+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:22.524 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:22.262+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:22.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:22.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:22.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:22.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:22.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:22.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:22.524 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:22.428+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:23.108 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:23.108 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:23.109 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.020+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:23.109 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.021+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:23.361 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:23.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:23.362 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:23.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:23.362 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:23.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:23.362 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:23.276+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:23.612 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:23.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:23.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:23.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:23.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:23.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:23.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:23.459+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:24.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:24.187+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14836 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.988+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:23.989+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:24.472+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:24.472+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:24.472+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:24.473+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:24.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:24.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:24.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:24.322+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:25.299 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:24.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:25.299 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:24.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:25.299 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:24.989+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:25.299 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:24.990+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:25.296+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:25.296+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:25.296+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:25.297+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:25.435+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:25.435+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:25.435+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:25.436+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:26.002+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:26.002+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:26.002+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:26.002+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:26.617 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:26.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:26.618 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:26.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:26.618 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:26.343+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:26.618 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:26.344+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:26.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:26.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:26.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:26.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:26.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:26.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:26.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:26.412+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:27.048+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:27.048+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:27.048+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:27.049+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:27.640 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:27.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:27.640 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:27.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:27.641 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:27.311+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:27.641 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:27.312+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:27.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:27.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:27.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:27.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:27.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:27.363+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:27.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:27.364+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:28.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:28.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:28.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:28.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:28.290+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:28.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:28.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:28.330+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:28.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:28.330+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:28.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:28.086+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:28.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:28.086+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:28.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:28.086+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:28.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:28.086+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:29.247 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:29.187+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14889 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:29.247 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:29.127+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:29.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:29.127+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:29.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:29.127+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:29.248 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:29.128+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:29.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:29.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:29.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:29.245+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:29.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:29.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:29.295+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:29.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:29.295+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:30.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:30.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:30.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:30.208+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:30.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:30.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:30.343+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:30.344+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:30.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:30.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:30.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:30.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:30.168+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:30.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:30.169+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:31.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:31.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:31.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:31.323+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:31.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:31.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:31.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:31.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:31.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:31.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:31.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:31.205+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:31.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:31.209+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:31.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:31.209+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:31.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:31.209+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:31.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:31.210+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:32.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:32.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:32.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:32.336+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:32.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:32.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:32.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:32.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:32.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:32.162+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:32.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:32.164+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:32.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:32.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:32.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:32.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:32.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:32.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:32.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:32.234+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:33.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:33.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:33.314 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:33.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:33.314 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:33.266+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:33.314 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:33.267+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:33.314 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:33.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:33.315 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:33.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:33.315 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:33.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:33.315 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:33.161+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:33.581 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:33.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:33.581 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:33.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:33.582 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:33.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:33.582 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:33.310+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:34.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:34.190+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14941 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:34.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:34.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:34.118+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:34.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:34.119+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:34.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:34.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:34.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:34.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:34.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:34.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:34.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:34.336+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:34.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:34.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:34.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:34.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:34.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:34.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:34.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:34.315+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:35.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:35.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:35.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:35.340+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:35.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:35.164+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:35.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:35.164+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:35.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:35.164+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:35.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:35.164+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:35.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:35.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:35.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:35.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:35.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:35.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:35.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:35.347+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:36.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:36.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:36.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:36.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:36.371+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:36.374+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:36.374+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:36.374+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:36.374+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:36.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:36.177+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:36.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:36.177+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:36.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:36.177+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:36.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:36.178+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:37.333+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:37.333+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:37.333+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:37.334+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:37.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:37.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:37.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:37.379+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:37.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:37.194+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:37.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:37.194+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:37.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:37.194+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:37.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:37.194+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:38.291+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:38.291+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:38.291+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:38.291+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:38.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:38.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:38.223+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:38.224+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:38.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:38.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:38.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:38.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:38.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:38.403+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:39.283 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:39.191+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 14994 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:39.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:39.199+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:39.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:39.199+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:39.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:39.199+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:39.284 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:39.200+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:39.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:39.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:39.448+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:39.449+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:39.281+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:39.281+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:39.281+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:39.281+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:40.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:40.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:40.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:40.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:40.462 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:40.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:40.463 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:40.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:40.463 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:40.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:40.463 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:40.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:40.463 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:40.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:40.463 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:40.160+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:40.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:40.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:40.459+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:40.460+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:41.490 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:41.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:41.490 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:41.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:41.490 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:41.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:41.490 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:41.136+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:41.490 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:41.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:41.491 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:41.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:41.491 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:41.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:41.491 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:41.336+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:41.487+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:41.487+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:41.487+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:41.488+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:42.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:42.100+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:42.100+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:42.100+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:42.755 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:42.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:42.755 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:42.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:42.756 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:42.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:42.756 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:42.530+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:42.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:42.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:42.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:42.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:42.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:42.360+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:42.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:42.361+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:43.325 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:43.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:43.325 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:43.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:43.325 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:43.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:43.326 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:43.068+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:43.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:43.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:43.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:43.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:43.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:43.518+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:43.614 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:43.518+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:43.614 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:43.322+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:43.615 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:43.322+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:43.615 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:43.322+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:43.615 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:43.323+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:44.541 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:44.193+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15046 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:44.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:44.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:44.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:44.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:44.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:44.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:44.542 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:44.102+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:44.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:44.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:44.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:44.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:44.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:44.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:44.543 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:44.285+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:44.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:44.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:44.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:44.540+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:45.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:45.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:45.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:45.105+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:45.800 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:45.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:45.800 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:45.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:45.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:45.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:45.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:45.569+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:45.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:45.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:45.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:45.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:45.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:45.320+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:45.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:45.321+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:46.325+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:46.325+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:46.325+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:46.326+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:46.135+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:46.135+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:46.135+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:46.136+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:46.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:46.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:46.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:46.590+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:47.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:47.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:47.359+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:47.359+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:47.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:47.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:47.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:47.185+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:47.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:47.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:47.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:47.627+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:48.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:48.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:48.344+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:48.345+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:48.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:48.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:48.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:48.153+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:48.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:48.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:48.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:48.668+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:49.305 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:49.194+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15099 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:49.306 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:49.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:49.306 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:49.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:49.306 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:49.160+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:49.306 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:49.161+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:49.303+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:49.303+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:49.303+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:49.304+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:49.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:49.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:49.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:49.699+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:50.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:50.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:50.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:50.330+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:50.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:50.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:50.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:50.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:50.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:50.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:50.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:50.161+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:50.730+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:51.358+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:51.358+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:51.358+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:51.359+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:51.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:51.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:51.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:51.143+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:51.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:51.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:51.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:51.768+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:52.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:52.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:52.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:52.324+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:52.140+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:52.140+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:52.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:52.140+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:52.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:52.141+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:52.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:52.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:52.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:52.810+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:53.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:53.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:53.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:53.145+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:53.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:53.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:53.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:53.351+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:53.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:53.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:53.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:54.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:53.786+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:54.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:54.195+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15151 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:54.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:54.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:54.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:54.302+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:54.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:54.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:54.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:54.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:54.111+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:55.141 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:54.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:55.141 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:54.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:55.141 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:54.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:55.141 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:54.827+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:55.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:55.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:55.287+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:55.287+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:55.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:55.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:55.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:55.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:55.139+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:55.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:55.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:55.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:55.782+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:56.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:56.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:56.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:56.182+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:56.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:56.183+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:56.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:56.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:56.318+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:56.319+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:57.196 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:56.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:57.196 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:56.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:57.197 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:56.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:57.197 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:56.825+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:57.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:57.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:57.193+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:57.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:57.194+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:57.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:57.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:57.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:57.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:57.344+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:57.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:57.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:57.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:57.797+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:58.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:58.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:58.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:58.196+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:58.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:58.197+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:58.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:58.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:58.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:58.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:58.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:58.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:58.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:58.393+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:50:59.200 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:58.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:59.200 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:58.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:59.200 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:58.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:50:59.200 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:58.827+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:59.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:50:59.197+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15203 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:50:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:59.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:59.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:59.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:59.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:59.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:50:59.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:50:59.234+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:50:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:59.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:59.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:59.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:59.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:50:59.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:50:59.362+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:00.216 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:59.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:00.217 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:59.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:00.217 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:59.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:00.217 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:50:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:50:59.827+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:00.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:00.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:00.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:00.324+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:00.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:00.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:00.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:00.214+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:01.230 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:00.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:01.230 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:00.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:01.230 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:00.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:01.231 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:00.814+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:01.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:01.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:01.227+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:01.228+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:01.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:01.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:01.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:01.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:01.285+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:01.772+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:01.772+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:01.772+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:01.772+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:02.241+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:02.241+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:02.241+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:02.242+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:02.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:02.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:02.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:02.275+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:02.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:02.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:02.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:02.745+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:03.235+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:03.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:03.235+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:03.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:03.235+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:03.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:03.236+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:03.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:03.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:03.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:03.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:03.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:03.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:03.564 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:03.314+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:03.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:03.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:03.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:03.790+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:04.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:04.198+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15256 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:04.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:04.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:04.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:04.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:04.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:04.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:04.344+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:04.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:04.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:04.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:04.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:04.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:04.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:04.187+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:05.147 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:04.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:05.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:04.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:05.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:04.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:05.148 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:04.829+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:05.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:05.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:05.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:05.145+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:05.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:05.145+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:05.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:05.303+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:05.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:05.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:05.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:05.304+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:05.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:05.305+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:05.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:05.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:05.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:05.807+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:06.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:06.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:06.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:06.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:06.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:06.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:06.554 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:06.175+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:06.555 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:06.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:06.555 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:06.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:06.555 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:06.316+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:06.555 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:06.317+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:06.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:06.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:06.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:06.765+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:07.146+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:07.146+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:07.146+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:07.147+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:07.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:07.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:07.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:07.328+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:07.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:07.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:07.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:07.721+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:08.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:08.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:08.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:08.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:08.173+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:08.173+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:08.173+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:08.173+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:08.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:08.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:08.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:08.726+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:09.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:09.200+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15308 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:09.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:09.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:09.278+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:09.278+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:09.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:09.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:09.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:09.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:09.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:09.217+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:09.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:09.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:09.720+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:09.721+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:10.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:10.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:10.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:10.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:10.317+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:10.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:10.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:10.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:10.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:10.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:10.210+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:10.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:10.211+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:10.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:10.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:10.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:10.727+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:11.202+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:11.202+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:11.202+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:11.203+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:11.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:11.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:11.267+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:11.268+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:11.691+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:11.691+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:11.691+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:11.692+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:12.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:12.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:12.190+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:12.190+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:12.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:12.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:12.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:12.265+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:12.992 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:12.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:12.993 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:12.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:12.993 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:12.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:12.993 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:12.690+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:13.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:13.206+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:13.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:13.206+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:13.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:13.206+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:13.259 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:13.207+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:13.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:13.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:13.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:13.257+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:13.732+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:13.732+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:13.732+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:13.733+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:14.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:14.201+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15361 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:14.290+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:14.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:14.290+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:14.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:14.290+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:14.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:14.291+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:14.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:14.169+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:14.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:14.169+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:14.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:14.169+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:14.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:14.169+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:14.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:14.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:14.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:14.745+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:15.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:15.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:15.259+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:15.260+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:15.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:15.188+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:15.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:15.188+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:15.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:15.188+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:15.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:15.188+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:15.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:15.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:15.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:15.734+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:16.283+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:16.283+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:16.283+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:16.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:16.284+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:16.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:16.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:16.152+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:16.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:16.153+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:16.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:16.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:16.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:17.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:16.767+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:17.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:17.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:17.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:17.312+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:17.313+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:17.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:17.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:17.156+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:17.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:17.156+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:17.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:17.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:17.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:17.799+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:18.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:18.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:18.184+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:18.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:18.184+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:18.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:18.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:18.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:18.333+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:19.173 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:18.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:19.173 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:18.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:19.174 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:18.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:19.174 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:18.824+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:19.203+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15413 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:19.171+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:19.171+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:19.171+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:19.171+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:19.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:19.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:19.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:19.379+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:20.134 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:19.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:20.134 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:19.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:20.135 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:19.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:20.135 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:19.858+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:20.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:20.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:20.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:20.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:20.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:20.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:20.385 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:20.132+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:20.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:20.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:20.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:20.383+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:21.111 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:20.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:21.111 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:20.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:21.111 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:20.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:21.111 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:20.810+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:21.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:21.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:21.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:21.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:21.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:21.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:21.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:21.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:21.782 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:21.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:21.783 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:21.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:21.783 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:21.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:21.783 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:21.410+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:21.780+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:21.780+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:21.780+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:21.780+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:22.450 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:22.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:22.451 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:22.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:22.451 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:22.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:22.451 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:22.145+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:22.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:22.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:22.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:22.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:22.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:22.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:22.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:22.449+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:22.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:22.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:22.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:22.807+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:23.462 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:23.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:23.462 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:23.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:23.463 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:23.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:23.463 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:23.100+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:23.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:23.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:23.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:23.461+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:24.140 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:23.845+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:24.140 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:23.845+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:24.140 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:23.845+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:24.140 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:23.846+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:24.472 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:24.204+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15466 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:24.472 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:24.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:24.472 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:24.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:24.472 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:24.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:24.473 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:24.138+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:24.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:24.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:24.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:24.470+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:25.162 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:24.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:25.162 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:24.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:25.163 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:24.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:25.163 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:24.817+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:25.437 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:25.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:25.438 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:25.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:25.438 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:25.159+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:25.438 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:25.161+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:25.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:25.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:25.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:25.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:25.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:25.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:25.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:25.435+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:25.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:25.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:25.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:25.783+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:26.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:26.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:26.185+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:26.186+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:26.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:26.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:26.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:26.389+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:26.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:26.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:26.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:26.761+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:27.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:27.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:27.208+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:27.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:27.209+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:27.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:27.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:27.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:27.398+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:27.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:27.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:27.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:27.721+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:28.446 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:28.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:28.446 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:28.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:28.446 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:28.252+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:28.446 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:28.253+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:28.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:28.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:28.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:28.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:28.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:28.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:28.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:28.444+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:28.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:28.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:28.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:28.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:28.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:29.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:29.206+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15517 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:29.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:29.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:29.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:29.208+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:29.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:29.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:29.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:29.472+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:29.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:29.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:29.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:29.694+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:30.477 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:30.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:30.478 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:30.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:30.478 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:30.163+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:30.478 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:30.163+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:30.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:30.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:30.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:30.476+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:30.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:30.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:30.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:30.676+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:31.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:31.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:31.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:31.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:31.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:31.132+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:31.482 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:31.133+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:31.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:31.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:31.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:31.628+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:31.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:31.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:31.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:31.480+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:32.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:32.093+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:32.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:32.093+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:32.450 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:32.093+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:32.450 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:32.094+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:32.659+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:32.659+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:32.659+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:32.660+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:32.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:32.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:32.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:32.447+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:33.448 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:33.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:33.448 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:33.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:33.448 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:33.071+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:33.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:33.072+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:33.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:33.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:33.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:33.616+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:33.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:33.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:33.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:33.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:33.446+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:34.485 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:34.207+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15570 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:34.485 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:34.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:34.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:34.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:34.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:34.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:34.486 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:34.066+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:34.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:34.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:34.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:34.666+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:34.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:34.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:34.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:34.483+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:35.494 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:35.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:35.494 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:35.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:35.494 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:35.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:35.494 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:35.070+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:35.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:35.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:35.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:35.492+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:35.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:35.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:35.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:35.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:35.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:35.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:35.814 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:35.715+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:36.475 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:36.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:36.475 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:36.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:36.476 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:36.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:36.476 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:36.096+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:36.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:36.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:36.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:36.667+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:36.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:36.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:36.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:36.473+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:37.052+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:37.052+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:37.052+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:37.054+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:37.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:37.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:37.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:37.622+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:37.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:37.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:37.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:37.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:38.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:38.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:38.054+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:38.055+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:38.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:38.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:38.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:38.612+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:38.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:38.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:38.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:38.439+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:39.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:39.209+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15622 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:39.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:39.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:39.092+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:39.093+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:39.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:39.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:39.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:39.447+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:39.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:39.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:39.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:39.634+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:40.448 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:40.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:40.448 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:40.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:40.448 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:40.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:40.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:40.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:40.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:40.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:40.680+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:40.680+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:40.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:40.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:40.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:40.446+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:41.424 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:41.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:41.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:41.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:41.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:41.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:41.425 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:41.138+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:41.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:41.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:41.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:41.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:41.734 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:41.422+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:41.734 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:41.422+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:41.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:41.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:41.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:41.731+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:42.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:42.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:42.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:42.410+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:42.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:42.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:42.161+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:42.161+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:42.750+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:42.750+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:42.750+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:42.751+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:43.172+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:43.172+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:43.172+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:43.173+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:43.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:43.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:43.393+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:43.393+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:43.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:43.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:43.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:43.801+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:44.358 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:44.210+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15675 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:44.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:44.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:44.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:44.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:44.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:44.207+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:44.359 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:44.208+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:44.804 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:44.355+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:44.804 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:44.355+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:44.804 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:44.355+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:44.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:44.356+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:44.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:44.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:44.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:44.802+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:45.203+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:45.203+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:45.203+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:45.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:45.204+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:45.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:45.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:45.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:45.362+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:46.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:45.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:46.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:45.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:46.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:45.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:46.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:45.818+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:46.490 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:46.345+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:46.490 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:46.345+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:46.490 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:46.345+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:46.490 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:46.345+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:46.491 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:46.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:46.491 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:46.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:46.491 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:46.216+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:46.491 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:46.217+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:47.236 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:46.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:47.236 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:46.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:47.236 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:46.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:47.237 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:46.849+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:47.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:47.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:47.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:47.302+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:47.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:47.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:47.233+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:47.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:47.234+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:48.244 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:47.897+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:48.244 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:47.897+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:48.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:47.897+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:48.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:47.898+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:48.241+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:48.241+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:48.241+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:48.242+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:48.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:48.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:48.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:48.333+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:49.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:48.874+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:49.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:48.874+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:49.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:48.874+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:49.215 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:48.875+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:49.211+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15727 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:49.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:49.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:49.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:49.357+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:49.279+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:49.279+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:49.279+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:49.280+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:49.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:49.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:49.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:49.848+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:50.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:50.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:50.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:50.385+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:50.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:50.319+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:50.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:50.319+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:50.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:50.319+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:50.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:50.320+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:51.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:50.839+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:51.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:50.839+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:51.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:50.839+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:51.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:50.840+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:51.298+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:51.298+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:51.298+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:51.299+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:51.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:51.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:51.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:51.353+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:51.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:51.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:51.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:51.807+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:52.594 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:52.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:52.595 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:52.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:52.595 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:52.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:52.595 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:52.311+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:52.595 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:52.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:52.595 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:52.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:52.595 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:52.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:52.595 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:52.319+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:52.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:52.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:52.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:52.798+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:53.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:53.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:53.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:53.310+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:53.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:53.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:53.352+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:53.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:53.353+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:53.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:53.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:53.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:53.807+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:54.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:54.213+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15780 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:54.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:54.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:54.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:54.383+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:54.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:54.336+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:54.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:54.336+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:54.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:54.336+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:54.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:54.337+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:55.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:54.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:55.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:54.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:55.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:54.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:55.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:54.834+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:55.304+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:55.304+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:55.304+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:55.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:55.305+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:55.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:55.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:55.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:55.421+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:55.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:55.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:55.809+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:55.810+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:56.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:56.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:56.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:56.433+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:56.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:56.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:56.322+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:56.323+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:57.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:56.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:57.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:56.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:57.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:56.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:57.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:56.816+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:57.304+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:57.304+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:57.304+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:57.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:57.304+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:57.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:57.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:57.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:57.462+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:57.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:57.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:57.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:57.790+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:58.778 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:58.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:58.779 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:58.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:58.779 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:58.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:58.779 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:58.340+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:58.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:58.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:58.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:58.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:58.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:58.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:58.780 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:58.506+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:51:59.217 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:58.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:59.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:58.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:59.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:58.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:51:59.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:58.812+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:59.520 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:51:59.214+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15832 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:51:59.520 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:59.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:59.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:59.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:59.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:59.344+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:51:59.521 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:51:59.345+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:51:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:59.517+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:59.517+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:59.517+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:51:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:51:59.518+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:59.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:59.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:59.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:51:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:51:59.841+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:00.357+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:00.357+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:00.357+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:00.358+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:00.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:00.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:00.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:00.481+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:00.861+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:00.861+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:00.861+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:00.862+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:01.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:01.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:01.309+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:01.309+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:01.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:01.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:01.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:01.461+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:01.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:01.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:01.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:01.867+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:02.345+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:02.346+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:02.346+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:02.346+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:02.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:02.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:02.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:02.480+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:02.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:02.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:02.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:02.828+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:03.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:03.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:03.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:03.384+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:03.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:03.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:03.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:03.491+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:04.218 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:03.829+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:04.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:03.829+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:04.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:03.829+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:04.219 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:03.830+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:04.498 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:04.216+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15885 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:04.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:04.422+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:04.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:04.422+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:04.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:04.422+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:04.499 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:04.423+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:04.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:04.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:04.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:04.496+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:04.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:04.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:04.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:04.818+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:05.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:05.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:05.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:05.462+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:05.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:05.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:05.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:05.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:05.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:05.424+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:05.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:05.425+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:05.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:05.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:05.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:05.867+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:06.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:06.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:06.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:06.475+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:06.395+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:06.395+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:06.395+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:06.395+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:06.850+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:06.850+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:06.850+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:06.850+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:07.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:07.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:07.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:07.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:07.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:07.349+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:07.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:07.349+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:07.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:07.349+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:07.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:07.350+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:08.305 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:07.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:08.305 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:07.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:08.305 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:07.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:08.306 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:07.836+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:08.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:08.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:08.302+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:08.303+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:08.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:08.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:08.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:08.464+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:09.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:08.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:09.220 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:08.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:09.221 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:08.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:09.221 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:08.814+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:09.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:09.217+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15937 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:09.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:09.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:09.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:09.416+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:09.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:09.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:09.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:09.336+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:09.855+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:09.855+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:09.855+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:09.856+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:10.465+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:10.465+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:10.465+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:10.466+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:10.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:10.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:10.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:10.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:10.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:10.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:10.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:10.368+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:10.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:10.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:10.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:10.880+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:11.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:11.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:11.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:11.494+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:11.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:11.332+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:11.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:11.332+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:11.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:11.332+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:11.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:11.333+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-12T03:52:11.994 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-03-12T03:52:11.994 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:52:11.994 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T03:52:12.013 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-12T03:52:12.013 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T03:52:12.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-12T03:52:12.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:11.925+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:12.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:11.925+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:12.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:11.925+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:12.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:11.926+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:12.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:12.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:12.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:12.471+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:12.298+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:12.298+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:12.298+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:12.298+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:13.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:12.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:13.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:12.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:13.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:12.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:13.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:12.916+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:13.291+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:13.291+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:13.291+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:13.292+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:13.499+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:13.499+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:13.499+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:13.499+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:14.222 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:13.904+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:14.223 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:13.904+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:14.223 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:13.904+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:14.223 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:13.904+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:14.545 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:14.219+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 15990 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:14.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:14.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:14.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:14.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:14.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:14.318+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:14.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:14.318+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:14.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:14.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:14.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:14.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:14.544+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:14.858+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:14.858+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:14.858+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:14.858+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:15.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:15.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:15.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:15.566+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:15.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:15.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:15.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:15.361+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:15.849+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:15.849+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:15.849+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:15.850+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:16.405+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:16.405+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:16.405+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:16.406+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:16.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:16.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:16.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:16.523+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:17.210 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:16.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:17.210 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:16.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:17.211 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:16.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:17.211 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:16.872+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:17.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:17.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:17.372+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:17.373+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:17.533+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:17.533+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:17.533+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:17.534+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:17.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:17.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:17.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:18.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:17.838+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:18.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:18.399+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:18.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:18.399+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:18.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:18.399+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:18.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:18.401+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:18.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:18.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:18.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:18.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:18.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:18.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:18.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:18.576+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:18.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:18.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:19.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:18.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:19.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:18.807+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:19.220+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16042 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:19.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:19.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:19.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:19.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:19.448+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:19.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:19.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:19.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:19.828+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:20.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:19.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:20.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:19.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:20.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:19.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:20.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:19.588+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:20.474+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:20.474+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:20.474+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:20.475+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:20.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:20.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:20.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:20.614+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:20.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:20.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:20.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:20.825+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:21.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:21.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:21.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:21.503+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:21.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:21.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:21.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:21.616+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:21.821+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:21.821+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:21.821+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:21.822+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:22.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:22.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:22.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:22.602+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:22.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:22.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:22.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:22.486+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:22.852+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:22.852+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:22.852+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:22.853+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:23.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:23.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:23.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:23.609+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:23.458+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:23.458+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:23.458+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:23.459+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:24.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:23.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:24.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:23.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:24.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:23.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:24.225 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:23.880+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:24.481 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:24.222+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16095 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:24.614+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:24.614+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:24.614+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:24.614+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:24.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:24.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:24.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:24.480+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:24.837+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:24.837+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:24.837+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:24.837+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:25.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:25.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:25.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:25.527+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:25.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:25.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:25.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:25.607+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:25.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:25.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:25.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:25.819+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:26.593+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:26.593+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:26.593+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:26.594+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:26.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:26.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:26.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:26.549+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:26.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:26.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:26.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:26.839+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:27.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:27.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:27.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:27.554+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:27.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:27.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:27.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:27.505+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:27.811+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:27.811+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:27.811+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:27.813+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:28.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:28.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:28.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:28.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:28.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:28.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:28.808 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:28.585+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:28.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:28.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:28.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:28.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:28.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:28.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:28.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:28.515+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:28.805+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:28.805+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:28.805+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:29.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:28.806+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:29.550 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:29.223+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16146 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:29.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:29.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:29.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:29.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:29.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:29.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:29.551 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:29.470+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:29.792+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:29.792+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:29.792+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:29.792+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:29.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:29.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:29.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:29.548+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:30.804 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:30.516+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:30.804 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:30.516+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:30.805 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:30.516+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:30.805 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:30.517+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:30.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:30.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:30.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:30.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:30.805 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:30.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:30.806 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:30.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:30.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:30.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:30.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:30.802+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:31.501+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:31.501+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:31.501+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:31.502+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:31.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:31.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:31.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:31.521+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:31.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:31.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:31.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:31.819+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:32.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:32.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:32.794 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:32.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:32.794 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:32.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:32.794 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:32.491+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:32.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:32.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:32.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:32.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:32.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:32.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:32.795 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:32.528+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:32.790+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:32.790+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:32.790+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:32.791+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:33.764 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:33.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:33.764 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:33.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:33.764 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:33.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:33.764 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:33.444+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:33.764 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:33.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:33.765 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:33.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:33.765 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:33.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:33.765 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:33.552+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:33.761+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:33.761+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:33.761+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:33.762+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:34.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:34.225+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16198 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:34.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:34.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:34.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:34.417+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:34.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:34.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:34.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:34.596+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:34.756+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:35.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:34.756+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:35.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:34.756+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:35.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:34.757+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:35.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:35.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:35.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:35.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:35.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:35.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:35.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:35.422+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:35.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:35.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:35.745 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:35.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:35.745 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:35.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:35.745 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:35.565+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:35.741+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:35.741+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:35.741+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:35.742+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:36.743 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:36.521+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:36.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:36.521+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:36.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:36.521+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:36.744 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:36.523+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:36.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:36.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:36.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:36.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:36.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:36.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:36.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:36.376+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:36.740+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:36.740+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:36.740+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:37.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:36.741+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:37.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:37.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:37.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:37.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:37.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:37.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:37.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:37.496+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:37.755 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:37.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:37.755 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:37.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:37.755 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:37.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:37.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:37.352+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:37.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:37.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:37.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:37.752+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:38.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:38.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:38.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:38.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:38.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:38.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:38.660 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:38.489+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:38.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:38.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:38.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:38.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:38.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:38.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:38.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:38.388+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:38.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:38.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:38.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:38.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:38.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:38.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:38.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:38.735+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:39.531 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:39.226+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16249 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:39.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:39.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:39.532 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:39.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:39.532 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:39.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:39.532 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:39.424+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:39.528+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:39.528+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:39.528+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:39.530+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:39.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:39.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:39.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:39.718+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:40.727 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:40.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:40.727 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:40.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:40.727 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:40.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:40.727 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:40.441+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:40.728 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:40.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:40.728 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:40.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:40.728 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:40.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:40.728 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:40.533+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:40.724+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:40.724+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:40.724+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:40.725+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:41.724 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:41.530+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:41.725 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:41.530+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:41.725 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:41.530+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:41.725 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:41.531+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:41.725 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:41.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:41.725 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:41.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:41.726 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:41.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:41.726 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:41.418+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:41.722+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:41.722+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:41.722+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:41.723+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:42.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:42.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:42.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:42.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:42.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:42.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:42.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:42.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:42.737 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:42.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:42.738 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:42.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:42.738 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:42.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:42.738 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:42.530+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:42.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:42.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:42.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:42.735+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:43.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:43.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:43.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:43.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:43.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:43.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:43.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:43.561+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:43.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:43.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:43.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:43.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:43.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:43.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:43.788 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:43.476+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:43.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:43.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:43.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:43.785+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:44.472 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:44.228+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16302 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:44.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:44.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:44.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:44.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:44.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:44.565+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:44.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:44.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:44.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:44.471+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:44.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:44.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:44.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:44.831+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:45.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:45.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:45.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:45.611+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:45.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:45.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:45.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:45.470+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:45.868+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:45.868+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:45.868+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:45.870+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:46.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:46.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:46.468+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:46.469+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:46.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:46.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:46.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:46.647+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:46.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:46.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:46.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:46.884+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:47.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:47.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:47.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:47.492+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:47.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:47.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:47.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:47.612+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:47.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:47.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:47.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:47.881+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:48.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:48.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:48.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:48.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:48.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:48.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:48.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:48.457+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:48.738 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:48.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:48.738 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:48.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:48.739 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:48.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:48.739 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:48.635+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:48.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:48.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:48.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:48.867+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:49.229+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16354 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:49.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:49.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:49.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:49.444+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:49.851+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:49.851+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:49.851+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:49.852+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:49.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:49.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:49.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:50.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:49.662+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:50.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:50.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:50.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:50.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:50.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:50.632+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:50.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:50.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:50.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:50.424+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:50.874+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:50.874+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:50.874+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:50.875+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:51.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:51.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:51.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:51.617+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:51.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:51.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:51.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:51.382+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:51.876+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:51.876+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:51.876+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:51.877+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:52.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:52.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:52.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:52.596+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:52.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:52.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:52.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:52.377+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:52.847+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:52.847+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:52.847+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:52.848+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:53.775 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:53.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:53.775 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:53.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:53.775 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:53.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:53.775 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:53.395+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:53.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:53.589+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:53.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:53.589+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:53.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:53.589+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:53.776 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:53.590+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:53.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:53.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:53.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:53.821+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:54.360 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:54.230+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16407 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:54.360 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:54.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:54.360 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:54.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:54.361 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:54.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:54.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:54.357+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:54.621+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:54.621+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:54.621+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:54.621+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:54.810+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:54.810+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:54.810+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:54.811+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:55.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:55.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:55.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:55.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:55.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:55.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:55.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:55.575+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:55.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:55.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:55.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:55.370+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:55.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:55.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:55.801 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:55.371+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:55.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:55.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:55.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:55.798+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:56.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:56.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:56.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:56.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:56.771 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:56.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:56.771 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:56.379+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:56.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:56.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:56.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:56.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:56.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:56.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:56.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:56.542+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:56.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:56.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:56.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:56.765+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:57.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:57.569+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:57.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:57.569+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:57.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:57.569+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:57.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:57.570+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:57.659 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:57.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:57.659 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:57.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:57.659 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:57.394+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:57.659 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:57.395+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:57.803+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:57.803+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:57.803+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:57.804+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:58.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:58.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:58.416+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:58.417+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:52:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:58.573+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:58.573+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:58.573+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:52:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:58.574+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:59.235 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:58.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:59.236 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:58.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:59.236 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:58.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:52:59.236 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:58.820+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:52:59.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:52:59.233+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16459 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:52:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:59.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:59.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:59.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:52:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:52:59.406+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:59.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:59.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:59.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:52:59.772+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:59.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:59.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:59.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:00.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:52:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:52:59.572+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:00.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:00.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:00.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:00.375+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:00.791+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:00.791+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:00.791+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:00.791+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:00.594+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:00.594+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:00.594+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:00.595+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:01.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:01.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:01.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:01.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:01.570+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:01.353+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:01.353+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:01.353+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:01.354+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:01.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:01.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:01.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:01.832+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:02.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:02.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:02.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:02.593+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:02.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:02.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:02.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:02.374+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:03.290 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:02.849+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:03.290 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:02.849+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:03.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:02.849+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:03.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:02.850+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:03.621 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:03.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:03.621 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:03.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:03.621 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:03.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:03.622 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:03.568+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:03.622 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:03.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:03.622 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:03.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:03.622 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:03.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:03.622 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:03.350+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:03.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:03.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:03.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:03.834+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:04.525 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:04.234+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16512 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:04.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:04.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:04.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:04.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:04.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:04.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:04.526 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:04.374+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:04.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:04.522+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:04.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:04.522+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:04.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:04.522+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:04.793 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:04.524+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:04.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:04.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:04.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:04.790+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:05.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:05.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:05.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:05.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:05.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:05.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:05.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:05.505+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:05.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:05.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:05.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:05.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:05.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:05.421+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:05.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:05.422+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:05.786+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:05.786+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:05.786+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:05.787+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:06.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:06.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:06.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:06.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:06.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:06.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:06.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:06.488+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:06.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:06.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:06.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:06.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:06.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:06.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:06.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:06.407+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:06.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:06.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:06.806+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:06.807+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:07.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:07.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:07.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:07.533+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:07.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:07.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:07.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:07.458+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:07.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:07.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:07.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:07.837+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:08.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:08.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:08.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:08.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:08.453+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:08.492+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:08.492+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:08.492+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:08.493+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:09.239 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:08.860+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:09.239 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:08.860+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:09.239 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:08.860+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:09.239 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:08.862+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:09.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:09.236+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16564 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:09.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:09.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:09.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:09.436+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:09.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:09.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:09.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:09.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:09.454+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:09.845+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:09.845+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:09.845+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:09.846+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:10.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:10.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:10.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:10.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:10.454+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:10.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:10.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:10.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:10.403+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:10.873+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:11.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:10.873+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:11.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:10.873+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:11.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:10.874+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:11.464+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:11.464+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:11.464+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:11.465+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:11.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:11.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:11.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:11.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:11.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:11.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:11.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:11.370+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:11.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:11.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:11.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:11.827+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:12.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:12.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:12.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:12.445+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:12.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:12.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:12.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:12.406+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:12.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:12.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:12.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:12.837+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:13.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:13.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:13.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:13.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:13.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:13.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:13.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:13.444+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:13.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:13.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:13.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:13.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:13.804 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:13.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:13.804 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:13.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:13.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:13.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:13.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:13.801+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:14.399 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:14.238+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16617 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:14.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:14.396+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:14.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:14.396+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:14.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:14.396+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:14.768 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:14.397+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:14.768 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:14.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:14.768 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:14.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:14.768 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:14.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:14.768 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:14.448+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:14.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:14.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:14.764+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:14.765+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:15.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:15.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:15.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:15.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:15.788 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:15.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:15.788 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:15.408+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:15.788 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:15.407+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:15.788 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:15.407+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:15.788 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:15.407+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:15.788 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:15.408+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:15.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:15.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:15.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:15.785+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:16.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:16.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:16.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:16.378+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:16.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:16.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:16.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:16.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:16.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:16.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:16.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:16.834+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:17.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:17.349+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:17.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:17.349+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:17.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:17.349+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:17.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:17.350+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:17.792 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:17.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:17.792 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:17.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:17.792 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:17.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:17.792 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:17.356+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:17.788+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:17.788+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:17.788+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:17.789+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:18.650 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:18.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:18.651 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:18.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:18.651 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:18.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:18.651 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:18.343+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:18.651 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:18.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:18.651 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:18.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:18.652 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:18.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:18.652 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:18.322+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:18.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:18.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:18.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:18.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:18.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:18.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:18.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:18.750+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:19.239+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16669 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:19.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:19.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:19.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:19.302+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:19.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:19.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:19.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:19.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:19.364+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:19.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:19.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:19.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:19.725+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:20.395+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:20.395+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:20.395+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:20.396+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:20.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:20.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:20.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:20.282+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:20.283+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:20.747+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:20.747+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:20.747+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:20.747+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:21.355+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:21.355+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:21.355+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:21.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:21.356+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:21.299+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:21.299+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:21.299+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:21.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:21.300+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:21.754+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:21.754+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:21.754+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:22.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:21.754+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:22.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:22.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:22.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:22.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:22.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:22.367+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:22.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:22.368+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:22.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:22.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:22.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:22.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:22.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:22.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:22.800 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:22.339+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:22.795+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:22.795+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:22.795+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:22.796+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:23.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:23.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:23.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:23.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:23.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:23.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:23.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:23.377+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:23.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:23.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:23.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:23.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:23.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:23.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:23.815 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:23.403+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:24.244 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:23.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:24.244 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:23.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:24.244 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:23.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:24.244 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:23.816+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:24.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:24.241+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16722 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:24.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:24.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:24.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:24.402+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:24.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:24.403+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:24.342+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:24.342+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:24.342+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:24.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:24.343+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:24.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:24.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:24.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:24.815+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:25.435+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:25.435+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:25.435+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:25.435+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:25.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:25.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:25.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:25.339+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:26.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:25.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:26.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:25.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:26.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:25.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:26.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:25.834+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:26.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:26.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:26.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:26.486+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:26.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:26.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:26.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:26.295+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:27.258 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:26.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:27.259 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:26.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:27.259 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:26.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:27.259 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:26.815+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:27.509 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:27.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:27.509 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:27.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:27.509 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:27.256+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:27.509 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:27.257+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:27.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:27.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:27.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:27.507+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:28.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:27.856+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:28.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:27.856+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:28.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:27.856+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:28.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:27.856+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:28.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:28.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:28.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:28.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:28.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:28.273+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:28.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:28.274+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:28.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:28.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:28.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:28.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:28.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:28.489+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:28.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:28.490+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:29.245 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:28.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:29.246 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:28.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:29.246 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:28.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:29.246 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:28.849+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:29.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:29.243+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16774 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:29.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:29.471+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:29.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:29.471+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:29.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:29.471+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:29.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:29.472+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:29.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:29.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:29.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:29.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:29.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:29.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:29.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:29.306+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:29.858+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:29.858+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:29.858+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:29.858+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:30.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:30.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:30.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:30.501+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:30.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:30.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:30.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:30.337+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:30.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:30.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:30.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:30.818+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:31.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:31.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:31.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:31.330+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:31.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:31.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:31.478+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:31.479+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:32.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:31.864+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:32.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:31.864+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:32.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:31.864+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:32.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:31.864+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:32.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:32.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:32.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:32.499+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:32.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:32.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:32.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:32.298+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:32.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:32.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:32.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:32.831+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:33.623 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:33.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:33.623 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:33.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:33.623 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:33.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:33.623 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:33.319+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:33.623 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:33.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:33.624 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:33.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:33.624 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:33.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:33.624 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:33.530+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:34.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:33.829+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:34.010 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:33.829+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:34.010 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:33.829+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:34.010 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:33.830+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:34.291 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:34.244+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16827 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:34.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:34.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:34.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:34.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:34.289+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:34.834 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:34.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:34.834 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:34.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:34.834 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:34.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:34.834 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:34.578+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:34.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:34.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:34.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:34.832+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:35.590 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:35.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:35.590 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:35.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:35.591 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:35.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:35.591 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:35.320+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:35.860 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:35.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:35.860 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:35.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:35.860 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:35.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:35.861 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:35.588+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:35.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:35.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:35.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:35.858+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:36.610 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:36.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:36.610 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:36.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:36.610 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:36.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:36.611 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:36.315+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:36.900 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:36.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:36.900 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:36.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:36.900 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:36.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:36.900 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:36.608+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:36.897+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:36.897+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:36.897+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:36.898+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:37.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:37.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:37.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:37.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:37.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:37.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:37.633 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:37.317+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:37.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:37.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:37.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:37.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:37.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:37.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:37.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:37.631+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:38.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:37.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:38.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:37.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:38.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:37.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:38.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:37.882+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:38.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:38.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:38.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:38.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:38.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:38.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:38.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:38.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:38.879 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:38.584+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:38.879 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:38.584+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:38.879 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:38.584+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:38.879 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:38.584+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:39.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:38.876+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:39.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:38.876+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:39.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:38.876+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:39.250 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:38.878+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:39.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:39.246+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16879 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:39.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:39.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:39.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:39.298+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:39.918 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:39.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:39.918 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:39.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:39.919 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:39.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:39.919 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:39.597+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:40.303 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:39.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:40.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:39.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:40.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:39.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:40.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:39.916+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:40.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:40.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:40.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:40.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:40.302+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:40.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:40.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:40.938 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:40.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:40.938 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:40.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:40.938 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:40.626+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:41.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:40.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:40.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:40.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:40.936+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:41.616 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:41.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:41.616 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:41.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:41.616 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:41.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:41.616 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:41.329+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:41.901 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:41.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:41.901 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:41.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:41.901 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:41.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:41.902 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:41.614+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:42.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:41.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:41.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:41.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:41.899+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:42.630 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:42.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:42.631 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:42.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:42.631 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:42.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:42.631 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:42.368+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:42.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:42.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:42.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:42.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:42.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:42.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:42.886 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:42.627+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:42.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:42.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:42.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:42.882+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:43.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:43.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:43.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:43.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:43.647+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:43.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:43.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:43.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:43.411+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:44.250 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:43.913+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:44.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:43.913+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:44.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:43.913+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:44.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:43.914+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:44.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:44.248+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16932 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:44.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:44.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:44.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:44.389+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:44.950 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:44.680+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:44.951 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:44.680+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:44.951 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:44.680+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:44.951 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:44.681+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:45.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:44.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:45.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:44.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:45.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:44.948+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:45.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:44.949+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:45.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:45.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:45.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:45.419+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:45.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:45.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:45.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:45.657+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:46.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:45.957+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:46.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:45.957+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:46.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:45.957+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:46.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:45.958+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:46.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:46.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:46.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:46.384+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:46.934 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:46.680+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:46.934 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:46.680+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:46.934 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:46.680+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:46.935 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:46.681+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:47.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:46.932+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:46.932+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:46.932+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:46.933+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:47.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:47.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:47.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:47.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:47.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:47.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:47.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:47.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:47.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:47.434+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:47.963 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:47.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:47.963 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:47.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:47.963 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:47.684+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:47.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:47.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:47.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:48.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:47.961+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:48.399+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:48.399+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:48.399+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:48.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:48.400+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:48.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:48.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:48.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:48.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:48.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:48.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:48.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:48.662+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:49.252 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:48.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:49.253 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:48.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:49.253 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:48.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:49.253 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:48.995+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:49.250+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 16984 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:49.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:49.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:49.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:49.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:49.425+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:50.038 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:49.706+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:50.039 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:49.706+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:50.039 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:49.706+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:50.039 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:49.706+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:50.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:50.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:50.036+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:50.036+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:50.735 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:50.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:50.736 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:50.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:50.736 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:50.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:50.736 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:50.388+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:51.017 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:50.733+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:51.017 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:50.733+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:51.017 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:50.733+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:51.017 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:50.734+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.014+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.014+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.014+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.015+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:51.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:51.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:51.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:51.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:51.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:51.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:51.771 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:51.415+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:51.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:51.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:51.768+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:51.768+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:51.974+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:52.730 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:52.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:52.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:52.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:52.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:52.418+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:52.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:52.419+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:53.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:52.728+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:53.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:52.728+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:53.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:52.728+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:53.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:52.729+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:53.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:53.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:53.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:53.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:53.008+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:53.718 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:53.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:53.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:53.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:53.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:53.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:53.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:53.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:54.023 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:53.716+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:54.024 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:53.716+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:54.024 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:53.716+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:54.024 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:53.717+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:54.311 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:54.251+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17036 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:54.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.022+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:54.768 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:54.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:54.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:54.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:54.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:54.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:54.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:54.374+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:54.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:54.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:54.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:54.766+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:54.992+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:55.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:55.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:55.734 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:55.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:55.734 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:55.354+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:55.734 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:55.355+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:55.734 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:55.731+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:55.734 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:55.731+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:55.734 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:55.731+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:55.732+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:55.949+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:55.949+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:55.949+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:55.949+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:56.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:56.308+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:56.308+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:56.308+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:56.906+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:56.906+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:56.906+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:56.906+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:56.763+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:57.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:56.763+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:57.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:56.763+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:57.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:56.764+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:57.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:57.333+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:57.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:57.333+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:57.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:57.333+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:57.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:57.334+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:57.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:57.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:57.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:57.873+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:57.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:57.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:57.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:57.739+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:58.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:58.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:58.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:58.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:58.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:58.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:58.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:58.362+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:53:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:58.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:58.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:58.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:53:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:58.742+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:59.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:58.825+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:59.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:58.825+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:59.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:58.825+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:53:59.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:58.826+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:53:59.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:53:59.253+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17089 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:53:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:59.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:59.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:59.319+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:53:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:53:59.319+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:59.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:59.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:59.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:53:59.747+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:59.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:59.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:59.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:53:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:53:59.786+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:00.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:00.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:00.307+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:00.308+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:00.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:00.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:00.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:00.737+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:00.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:00.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:00.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:00.786+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:01.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:01.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:01.288+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:01.289+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:01.748+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:01.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:01.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:01.750+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:01.788+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:02.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:01.788+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:02.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:01.788+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:02.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:01.789+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:02.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:02.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:02.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:02.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:02.748 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:02.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:02.748 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:02.333+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:02.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:02.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:02.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:02.745+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:02.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:02.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:02.827+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:02.828+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:03.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:03.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:03.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:03.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:03.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:03.317+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:03.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:03.318+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:03.748+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:03.748+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:03.748+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:03.749+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:03.861+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:04.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:03.861+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:04.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:03.861+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:04.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:03.862+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:04.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:04.254+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17140 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:04.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:04.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:04.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:04.285+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:04.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:04.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:04.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:04.799+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:04.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:04.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:04.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:04.820+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:05.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:05.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:05.305+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:05.305+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:05.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:05.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:05.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:05.836+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:05.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:05.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:05.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:05.842+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:06.796 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:06.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:06.796 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:06.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:06.796 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:06.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:06.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:06.350+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:06.794+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:06.794+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:06.794+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:06.794+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:06.828+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:06.828+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:07.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:06.828+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:07.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:06.829+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:07.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:07.326+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:07.825 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:07.326+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:07.825 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:07.326+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:07.825 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:07.327+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:07.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:07.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:07.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:07.831+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:08.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:07.807+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:07.807+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:07.807+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:07.807+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:08.761 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:08.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:08.761 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:08.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:08.761 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:08.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:08.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:08.315+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:09.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:08.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:09.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:08.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:09.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:08.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:09.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:08.759+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:08.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:09.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:08.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:09.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:08.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:09.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:08.858+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:09.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:09.256+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17193 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:09.358+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:09.358+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:09.358+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:09.359+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:09.899+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:09.899+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:09.899+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:09.900+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:09.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:09.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:10.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:09.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:10.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:09.747+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:10.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:10.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:10.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:10.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:10.718 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:10.371+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:10.718 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:10.371+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:10.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:10.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:10.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:10.863+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:10.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:10.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:10.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:10.715+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:11.691 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:11.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:11.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:11.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:11.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:11.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:11.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:11.342+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:11.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:11.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:11.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:11.690+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:11.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:11.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:11.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:11.906+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-12T03:54:12.143 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-03-12T03:54:12.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:54:12.144 DEBUG:teuthology.orchestra.run.smithi118:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-03-12T03:54:12.181 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc: Cannot mkdir: Input/output error 2024-03-12T03:54:12.181 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3: Cannot mkdir: Input/output error 2024-03-12T03:54:12.182 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/random_range.3: Cannot open: Input/output error 2024-03-12T03:54:12.182 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/str_to_bytes.3: Cannot open: Input/output error 2024-03-12T03:54:12.182 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/usctest.3: Cannot open: Input/output error 2024-03-12T03:54:12.182 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/string_to_tokens.3: Cannot open: Input/output error 2024-03-12T03:54:12.182 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/Makefile: Cannot open: Input/output error 2024-03-12T03:54:12.182 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/pattern.3: Cannot open: Input/output error 2024-03-12T03:54:12.182 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/tst_tmpdir.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/forker.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/rmobj.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/random_range_seed.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/parse_open_flags.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/parse_opts.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/write_log.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/parse_ranges.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/tst_set_error.3: Cannot open: Input/output error 2024-03-12T03:54:12.183 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/tst_res.3: Cannot open: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/tst_sig.3: Cannot open: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man3/get_attrib.3: Cannot open: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man1: Cannot mkdir: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man1/ltp-bump.1: Cannot open: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man1/doio.1: Cannot open: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man1/Makefile: Cannot open: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man1/ltp-pan.1: Cannot open: Input/output error 2024-03-12T03:54:12.184 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/man1/iogen.1: Cannot open: Input/output error 2024-03-12T03:54:12.185 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/testcases: Cannot mkdir: Input/output error 2024-03-12T03:54:12.186 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/testcases/kernel.txt: Cannot open: Input/output error 2024-03-12T03:54:12.186 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/testcases/misc.txt: Cannot open: Input/output error 2024-03-12T03:54:12.186 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/testcases/network.txt: Cannot open: Input/output error 2024-03-12T03:54:12.186 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/testcases/commands.txt: Cannot open: Input/output error 2024-03-12T03:54:12.186 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/testcases/open_posix_testsuite.txt: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/testcases/open_hpi_testsuite.txt: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/ltp-howto.lyx: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/Makefile: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/automation-cookbook.html: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/examples: Cannot mkdir: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/examples/quickhit.c: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/examples/sudoers: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/examples/super.tab: Cannot open: Input/output error 2024-03-12T03:54:12.187 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/ltp-run-files.txt: Cannot open: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/tpm-commands.txt: Cannot open: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/ltp-howto.txt: Cannot open: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/network_stress.txt: Cannot open: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/doc/automation-README.html: Cannot open: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4: Cannot mkdir: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-cgroupstats.m4: Cannot open: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-signalfd.m4: Cannot open: Input/output error 2024-03-12T03:54:12.188 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/GNUmakefile: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-eventfd.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-quota.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-version.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-cap.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-selinux.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-signal.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-ptrace.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-numa.m4: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/Makefile.in: Cannot open: Input/output error 2024-03-12T03:54:12.189 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-modify_ldt.m4: Cannot open: Input/output error 2024-03-12T03:54:12.190 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-unshare.m4: Cannot open: Input/output error 2024-03-12T03:54:12.190 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-taskstats.m4: Cannot open: Input/output error 2024-03-12T03:54:12.190 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/ltp-crypto.m4: Cannot open: Input/output error 2024-03-12T03:54:12.190 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/m4/Makefile.am: Cannot open: Input/output error 2024-03-12T03:54:12.190 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/runltp: Cannot open: Input/output error 2024-03-12T03:54:12.190 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/Makefile: Cannot open: Input/output error 2024-03-12T03:54:12.191 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi118.front.sepia.ceph.com 2024-03-12T03:54:12.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-12T03:54:12.192 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-12T03:54:12.195 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/configure: Cannot open: Input/output error 2024-03-12T03:54:12.195 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/config.guess: Cannot open: Input/output error 2024-03-12T03:54:12.195 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/execltp.in: Cannot open: Input/output error 2024-03-12T03:54:12.195 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib: Cannot mkdir: Input/output error 2024-03-12T03:54:12.195 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/rmobj.c: Cannot open: Input/output error 2024-03-12T03:54:12.195 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/datapid.c: Cannot open: Input/output error 2024-03-12T03:54:12.195 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/tst_kvercmp.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/tst_sig.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/tst_cwd_has_free.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/parse_opts.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/tlibio.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/cloner.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/system_specific_process_info.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/tst_tmpdir.c: Cannot open: Input/output error 2024-03-12T03:54:12.196 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/open_flags.c: Cannot open: Input/output error 2024-03-12T03:54:12.197 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/get_high_address.c: Cannot open: Input/output error 2024-03-12T03:54:12.197 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/Makefile: Cannot open: Input/output error 2024-03-12T03:54:12.197 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/write_log.c: Cannot open: Input/output error 2024-03-12T03:54:12.197 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/file_lock.c: Cannot open: Input/output error 2024-03-12T03:54:12.197 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/tst_is_cwd.c: Cannot open: Input/output error 2024-03-12T03:54:12.197 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/string_to_tokens.c: Cannot open: Input/output error 2024-03-12T03:54:12.197 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/system_specific_hugepages_info.c: Cannot open: Input/output error 2024-03-12T03:54:12.198 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/str_to_bytes.c: Cannot open: Input/output error 2024-03-12T03:54:12.198 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/random_range.c: Cannot open: Input/output error 2024-03-12T03:54:12.198 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-12T03:54:12.198 INFO:tasks.workunit.client.0.smithi118.stderr:gzip: tar: ltp-full-20091231/lib/libtestsuite.c: Cannot open: Input/output error 2024-03-12T03:54:12.198 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/ltp.pc.in: Cannot open: Input/output error 2024-03-12T03:54:12.198 INFO:tasks.workunit.client.0.smithi118.stderr:stdin: Stale file handle 2024-03-12T03:54:12.198 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231/lib/tst_res.c: Cannot open: Input/output error 2024-03-12T03:54:12.199 INFO:tasks.workunit.client.0.smithi118.stderr:tar: Unexpected EOF in archive 2024-03-12T03:54:12.199 INFO:tasks.workunit.client.0.smithi118.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-03-12T03:54:12.199 INFO:tasks.workunit.client.0.smithi118.stderr:tar: Error is not recoverable: exiting now 2024-03-12T03:54:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-12T03:54:12.199 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-12T03:54:12.200 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-12T03:54:12.264 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-12T03:54:12.264 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-12T03:54:12.533 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:12.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:12.534 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:12.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:12.534 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:12.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:12.534 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:12.329+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:12.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:12.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:12.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:12.693+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:12.887+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:12.887+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:12.887+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:12.888+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:13.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:13.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:13.625 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:13.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:13.625 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:13.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:13.625 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:13.333+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:13.945 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:13.844+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:13.945 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:13.844+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:13.946 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:13.844+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:13.946 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:13.845+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:13.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:13.665+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:13.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:13.665+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:13.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:13.665+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:13.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:13.666+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:14.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:14.257+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17245 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:14.303+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:14.303+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:14.303+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:14.304+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:14.869 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:14.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:14.869 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:14.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:14.869 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:14.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:14.870 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:14.657+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:15.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:14.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:15.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:14.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:15.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:14.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:15.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:14.868+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:15.295+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:15.295+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:15.295+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:15.296+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:15.912 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:15.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:15.912 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:15.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:15.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:15.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:15.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:15.656+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:15.910+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:15.910+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:15.910+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:15.910+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:16.628 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:16.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:16.629 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:16.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:16.629 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:16.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:16.629 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:16.336+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:16.882 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:16.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:16.882 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:16.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:16.882 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:16.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:16.883 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:16.627+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:17.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:16.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:16.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:16.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:16.880+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:17.605 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:17.349+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:17.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:17.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:17.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:17.350+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:17.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:17.350+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:17.865 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:17.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:17.865 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:17.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:17.866 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:17.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:17.866 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:17.604+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:17.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:17.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:17.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:17.864+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:18.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:18.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:18.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:18.561+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:18.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:18.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:18.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:18.381+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:19.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:18.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:19.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:18.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:19.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:18.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:19.262 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:18.829+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:19.546 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:19.259+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17296 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:19.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:19.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:19.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:19.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:19.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:19.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:19.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:19.425+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:19.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:19.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:19.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:19.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:19.545+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:19.864+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:19.864+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:19.864+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:19.865+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:20.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:20.535+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:20.535+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:20.535+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:20.536+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:20.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:20.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:20.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:20.461+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:21.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:20.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:20.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:20.838+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:20.838+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:21.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:21.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:21.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:21.559+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:21.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:21.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:21.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:21.444+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:21.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:21.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:21.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:21.862+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:22.512+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:22.512+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:22.512+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:22.513+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:22.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:22.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:22.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:22.423+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:23.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:22.875+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:22.875+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:22.875+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:22.875+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:23.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:23.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:23.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:23.443+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:23.444+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:23.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:23.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:23.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:23.542+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:24.264 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:23.919+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:24.264 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:23.919+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:24.264 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:23.919+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:24.264 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:23.919+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:24.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:24.261+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17348 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:24.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:24.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:24.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:24.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:24.486+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:24.932 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:24.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:24.932 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:24.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:24.933 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:24.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:24.933 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:24.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:24.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:24.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:24.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:24.930+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:25.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:25.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:25.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:25.498+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:25.539+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:25.539+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:25.539+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:25.540+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:26.233 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:25.959+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:26.233 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:25.959+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:26.234 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:25.959+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:26.234 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:25.959+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:26.510 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:26.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:26.510 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:26.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:26.510 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:26.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:26.510 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:26.476+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:26.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:26.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:26.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:26.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:26.509+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:27.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:26.982+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:26.982+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:26.982+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:26.983+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:27.511+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:27.511+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:27.511+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:27.512+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:27.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:27.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:27.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:27.450+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:28.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.009+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:28.547+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:28.547+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:28.547+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:28.548+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:28.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:28.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:28.440+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:28.441+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:29.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:29.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:29.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:29.265 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:28.974+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:29.537 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:29.262+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17401 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:29.537 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:29.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:29.537 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:29.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:29.537 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:29.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:29.538 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:29.396+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:29.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:29.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:29.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:29.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:29.535+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:30.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:29.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:29.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:29.967+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:29.967+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:30.349+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:30.349+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:30.349+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:30.349+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:30.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:30.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:30.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:30.558+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:31.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:30.998+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:30.998+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:30.998+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:30.998+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:31.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:31.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:31.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:31.600+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:31.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:31.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:31.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:31.363+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:31.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:31.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:31.996+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:31.997+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:32.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:32.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:32.329+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:32.330+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:32.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:32.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:32.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:32.572+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:33.293 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:33.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:33.293 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:33.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:33.293 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:33.033+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:33.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:33.033+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:33.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:33.290+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:33.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:33.290+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:33.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:33.290+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:33.547 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:33.291+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:33.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:33.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:33.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:33.545+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:34.268 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.013+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:34.268 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.013+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:34.268 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.013+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:34.268 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.015+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:34.547 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:34.264+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17453 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:34.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:34.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:34.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:34.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:34.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:34.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:34.548 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:34.333+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:34.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:34.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:34.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:34.551+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.971+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:34.972+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:35.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:35.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:35.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:35.571+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:35.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:35.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:35.335+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:35.335+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.009+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.011+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:36.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:36.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:36.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:36.578+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:36.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:36.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:36.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:36.364+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:36.969+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:37.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:37.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:37.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:37.576+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:37.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:37.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:37.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:37.340+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:38.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:37.929+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:37.929+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:37.929+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:37.929+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:38.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:38.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:38.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:38.343+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:38.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:38.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:38.529+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:38.530+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:39.269 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:38.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:39.270 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:38.937+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:39.270 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:38.938+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:39.270 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:38.938+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:39.559 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:39.266+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17506 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:39.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:39.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:39.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:39.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:39.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:39.297+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:39.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:39.298+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:39.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:39.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:39.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:39.557+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:40.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:39.892+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:39.892+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:39.892+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:39.893+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:40.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:40.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:40.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:40.514+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:40.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:40.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:40.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:40.337+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:41.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:40.895+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:40.895+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:40.895+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:40.895+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:41.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:41.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:41.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:41.484+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:41.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:41.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:41.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:41.342+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:42.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:41.860+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:41.860+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:41.860+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:41.861+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:42.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:42.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:42.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:42.501+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:42.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:42.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:42.372+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:42.373+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:42.878+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:42.878+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:42.878+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:42.879+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:43.626 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:43.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:43.626 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:43.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:43.626 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:43.340+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:43.627 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:43.341+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:43.627 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:43.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:43.627 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:43.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:43.627 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:43.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:43.627 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:43.509+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:44.270 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:43.918+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:44.271 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:43.918+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:44.271 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:43.918+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:44.271 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:43.919+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:44.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:44.267+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17558 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:44.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:44.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:44.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:44.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:44.339+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:44.491+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:44.491+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:44.491+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:44.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:44.492+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:44.943+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:44.943+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:44.943+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:44.943+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:45.322+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:45.322+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:45.322+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:45.323+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:45.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:45.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:45.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:45.489+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:46.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:45.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:46.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:45.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:46.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:45.983+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:46.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:45.984+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:46.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:46.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:46.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:46.451+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:46.452+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:46.296+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:46.296+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:46.296+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:46.297+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:47.268 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:46.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:47.268 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:46.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:47.268 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:46.960+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:47.269 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:46.961+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:47.443+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:47.443+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:47.443+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:47.444+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:47.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:47.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:47.265+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:47.267+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:48.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:47.916+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:48.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:47.916+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:48.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:47.916+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:48.255 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:47.917+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:48.252+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:48.252+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:48.252+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:48.253+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:48.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:48.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:48.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:48.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:48.447+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:49.263 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:48.945+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:49.264 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:48.945+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:49.264 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:48.945+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:49.264 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:48.946+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:49.269+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17611 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:49.432+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:49.432+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:49.432+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:49.433+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:49.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:49.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:49.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:49.262+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:50.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:49.981+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:50.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:49.981+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:50.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:49.981+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:50.309 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:49.981+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:50.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:50.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:50.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:50.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:50.448+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:50.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:50.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:50.306+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:50.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:50.974+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:50.974+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:50.974+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:50.975+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:51.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:51.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:51.338+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:51.339+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:51.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:51.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:51.470+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:51.471+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:52.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:51.951+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:51.951+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:51.951+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:51.952+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:52.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:52.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:52.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:52.497+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:52.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:52.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:52.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:52.376+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:53.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:52.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:52.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:52.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:53.000+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:53.545+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:53.545+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:53.545+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:53.547+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:53.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:53.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:53.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:53.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:53.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:53.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:53.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:53.425+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:54.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:53.984+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:54.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:53.984+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:54.274 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:53.984+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:54.274 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:53.985+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:54.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:54.271+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17663 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:54.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:54.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:54.378+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:54.379+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:55.006 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:54.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:55.007 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:54.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:55.007 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:54.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:55.007 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:54.597+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:55.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.005+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:55.611 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:55.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:55.612 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:55.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:55.612 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:55.356+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:55.612 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:55.358+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:55.991 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:55.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:55.992 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:55.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:55.992 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:55.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:55.992 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:55.610+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:56.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:55.990+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:56.652 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:56.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:56.652 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:56.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:56.652 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:56.361+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:56.652 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:56.362+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:56.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:56.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:56.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:56.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:56.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:56.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:56.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:56.650+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:57.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:56.942+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:56.942+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:56.942+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:56.943+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:57.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:57.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:57.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:57.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:57.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:57.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:57.637 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:57.342+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:57.917 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:57.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:57.917 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:57.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:57.917 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:57.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:57.918 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:57.635+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:58.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:57.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:57.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:57.915+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:58.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:57.916+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:58.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:58.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:58.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:58.594+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:58.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:58.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:58.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:58.388+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:59.275 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:58.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:59.275 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:58.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:59.275 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:58.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:54:59.275 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:58.898+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:54:59.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:54:59.272+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17716 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:54:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:59.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:59.434+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:59.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:54:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:54:59.436+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:54:59.874 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:59.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:59.874 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:59.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:59.874 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:59.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:54:59.874 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:54:59.560+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:00.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:59.871+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:59.871+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:59.871+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:54:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:54:59.872+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:00.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:00.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:00.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:00.532+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:00.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:00.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:00.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:00.465+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:01.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:00.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:00.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:00.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:00.827+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:01.783 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:01.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:01.783 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:01.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:01.783 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:01.487+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:01.784 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:01.488+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:01.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:01.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:01.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:01.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:01.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:01.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:01.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:01.492+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:02.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:01.780+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:01.780+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:01.780+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:01.781+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:02.785 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:02.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:02.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:02.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:02.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:02.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:02.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:02.528+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:02.786 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:02.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:02.786 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:02.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:02.786 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:02.502+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:02.787 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:02.503+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:02.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:02.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:02.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:02.784+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:03.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:03.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:03.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:03.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:03.564+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:03.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:03.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:03.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:03.494+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:04.276 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:03.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:04.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:03.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:04.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:03.820+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:04.277 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:03.821+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:04.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:04.274+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17768 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:04.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:04.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:04.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:04.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:04.457+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:04.844 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:04.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:04.844 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:04.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:04.844 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:04.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:04.845 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:04.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:04.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:04.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:04.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:04.842+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:05.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:05.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:05.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:05.472+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:05.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:05.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:05.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:05.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:05.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:05.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:05.582+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:06.169 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:05.854+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:06.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:05.854+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:06.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:05.854+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:06.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:05.855+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:06.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:06.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:06.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:06.593+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:06.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:06.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:06.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:06.514+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:06.865+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:06.865+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:06.865+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:06.866+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:07.624+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:07.624+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:07.624+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:07.624+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:07.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:07.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:07.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:07.526+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:08.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:07.885+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:07.885+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:07.885+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:07.885+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:08.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:08.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:08.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:08.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:08.636+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:08.487+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:08.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:08.487+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:08.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:08.487+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:08.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:08.487+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:09.278 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:08.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:09.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:08.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:09.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:08.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:09.291 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:08.878+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:09.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:09.276+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17821 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:09.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:09.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:09.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:09.447+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:09.637+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:09.637+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:09.637+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:09.638+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:09.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:09.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:09.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:09.844+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:10.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:10.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:10.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:10.609+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:10.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:10.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:10.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:10.438+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:11.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:10.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:11.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:10.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:11.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:10.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:11.314 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:10.813+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:11.795 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:11.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:11.796 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:11.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:11.796 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:11.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:11.796 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:11.425+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:11.796 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:11.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:11.796 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:11.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:11.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:11.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:11.797 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:11.594+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:12.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:11.793+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:11.793+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:11.793+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:11.794+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:12.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:12.637+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:12.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:12.637+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:12.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:12.637+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:12.772 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:12.638+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:12.772 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:12.398+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:12.772 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:12.398+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:12.772 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:12.398+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:12.772 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:12.399+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:12.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:12.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:12.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:12.770+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:13.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:13.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:13.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:13.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:13.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:13.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:13.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:13.658+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:13.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:13.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:13.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:13.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:13.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:13.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:13.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:13.424+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:13.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:13.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:13.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:13.729+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:14.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:14.277+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17873 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:14.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:14.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:14.452+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:14.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:14.453+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:14.860 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:14.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:14.860 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:14.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:14.860 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:14.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:14.860 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:14.662+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:14.861 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:14.775+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:14.861 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:14.775+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:14.861 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:14.775+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:14.861 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:14.776+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:15.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:15.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:15.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:15.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:15.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:15.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:15.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:15.481+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:15.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:15.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:15.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:15.652+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:15.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:15.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:15.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:15.787+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:16.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:16.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:16.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:16.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:16.670+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:16.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:16.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:16.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:16.466+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:17.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:16.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:16.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:16.815+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:16.815+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:17.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:17.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:17.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:17.670+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:17.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:17.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:17.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:17.429+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:18.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:17.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:17.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:17.831+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:17.831+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:18.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:18.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:18.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:18.479+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:18.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:18.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:18.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:18.635+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:19.281 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:18.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:19.282 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:18.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:19.282 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:18.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:19.282 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:18.880+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:19.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:19.278+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17926 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:19.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:19.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:19.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:19.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:19.474+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:19.899 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:19.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:19.899 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:19.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:19.899 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:19.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:19.899 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:19.863+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:19.899 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:19.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:19.900 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:19.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:19.900 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:19.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:19.900 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:19.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:20.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:20.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:20.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:20.624+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:20.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:20.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:20.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:20.501+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:21.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:20.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:20.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:20.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:20.866+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:21.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:21.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:21.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:21.618+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:21.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:21.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:21.457+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:21.457+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:21.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:21.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:21.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:21.873+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:22.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:22.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:22.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:22.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:22.588+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:22.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:22.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:22.455+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:22.456+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:22.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:22.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:22.824+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:22.825+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:23.718 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:23.578+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:23.719 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:23.578+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:23.719 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:23.578+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:23.719 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:23.579+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:23.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:23.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:23.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:23.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:23.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:23.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:23.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:23.455+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:24.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:23.869+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:23.869+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:23.869+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:23.870+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:24.403 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:24.281+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 17977 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:24.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:24.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:24.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:24.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:24.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:24.452+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:24.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:24.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:24.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:24.598+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:25.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:24.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:24.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:24.826+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:24.827+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:25.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:25.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:25.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:25.440+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:25.555+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:25.555+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:25.555+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:25.556+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:25.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:25.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:25.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:25.868+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:26.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:26.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:26.432+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:26.433+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:26.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:26.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:26.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:26.586+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:26.823+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:26.823+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:26.823+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:26.823+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:27.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:27.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:27.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:27.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:27.408+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:27.552+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:27.552+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:27.552+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:27.552+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:28.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:27.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:27.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:27.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:27.842+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:28.511+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:28.511+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:28.511+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:28.512+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:28.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:28.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:28.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:28.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:29.285 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:28.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:29.285 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:28.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:29.285 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:28.819+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:29.286 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:28.819+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:29.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:29.283+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18030 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:29.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:29.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:29.485+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:29.485+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:29.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:29.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:29.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:29.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:29.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:29.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:29.862+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:29.862+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:30.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:30.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:30.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:30.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:30.433+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:30.465+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:30.465+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:30.465+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:30.466+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:31.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:30.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:30.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:30.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:30.843+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:31.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:31.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:31.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:31.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:31.478+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:31.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:31.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:31.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:31.445+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:31.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:31.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:31.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:31.848+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:32.503+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:32.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:32.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:32.505+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:32.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:32.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:32.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:32.429+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:33.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:32.846+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:32.846+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:32.846+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:32.847+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:33.456+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:33.456+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:33.456+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:33.457+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:33.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:33.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:33.392+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:33.393+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:34.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:33.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:34.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:33.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:34.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:33.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:34.273 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:33.836+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:34.538 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:34.284+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18082 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:34.539 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:34.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:34.539 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:34.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:34.539 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:34.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:34.539 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:34.397+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:34.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:34.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:34.540 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:34.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:34.540 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:34.453+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:34.540 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:34.454+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:35.136 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:34.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:35.136 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:34.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:35.136 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:34.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:35.136 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:34.818+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:35.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:35.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:35.444+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:35.445+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:35.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:35.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:35.500+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:35.501+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:36.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:35.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:35.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:35.843+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:35.844+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:36.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:36.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:36.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:36.442+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:36.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:36.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:36.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:36.537+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:37.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:36.825+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:36.825+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:36.825+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:36.825+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:37.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:37.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:37.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:37.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:37.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:37.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:37.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:37.533+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:37.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:37.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:37.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:37.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:37.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:37.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:37.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:37.493+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:38.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:37.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:37.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:37.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:37.790+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:38.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:38.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:38.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:38.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:38.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:38.464+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:38.784 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:38.465+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:38.784 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:38.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:38.785 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:38.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:38.785 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:38.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:38.785 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:38.508+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:39.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:38.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:39.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:38.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:39.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:38.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:39.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:38.782+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:39.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:39.286+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18135 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:39.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:39.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:39.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:39.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:39.463+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:39.495+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:39.495+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:39.495+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:39.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:39.496+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:40.052 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:39.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:40.053 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:39.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:40.053 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:39.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:40.053 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:39.801+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:40.765 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:40.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:40.766 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:40.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:40.766 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:40.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:40.766 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:40.459+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:40.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:40.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:40.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:40.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:40.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:40.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:40.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:40.499+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:40.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:40.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:40.762+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:40.763+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:41.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:41.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:41.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:41.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:41.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:41.480+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:41.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:41.481+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:41.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:41.547+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:41.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:41.547+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:41.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:41.547+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:41.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:41.547+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:41.794+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:41.794+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:41.794+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:41.795+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:42.784 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:42.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:42.785 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:42.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:42.785 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:42.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:42.785 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:42.572+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:42.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:42.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:42.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:42.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:42.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:42.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:42.786 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:42.439+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:43.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:42.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:42.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:42.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:42.783+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:43.754 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:43.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:43.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:43.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:43.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:43.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:43.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:43.528+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:43.755 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:43.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:43.755 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:43.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:43.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:43.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:43.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:43.408+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:44.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:43.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:43.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:43.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:43.753+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:44.433 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:44.288+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18187 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:44.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:44.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:44.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:44.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:44.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:44.431+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:44.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:44.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:44.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:44.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:44.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:44.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:44.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:44.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:44.733 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:44.550+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:44.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:44.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:44.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:44.729+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:45.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:45.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:45.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:45.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:45.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:45.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:45.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:45.532+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:45.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:45.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:45.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:45.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:45.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:45.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:45.733 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:45.384+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:46.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:45.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:45.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:45.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:45.730+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:46.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:46.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:46.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:46.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:46.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:46.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:46.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:46.412+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:46.717 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:46.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:46.718 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:46.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:46.718 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:46.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:46.718 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:46.571+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:47.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:46.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:46.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:46.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:46.715+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:47.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:47.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:47.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:47.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:47.756 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:47.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:47.756 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:47.537+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:47.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:47.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:47.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:47.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:47.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:47.417+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:47.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:47.417+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:48.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:47.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:47.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:47.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:47.753+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:48.756 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:48.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:48.757 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:48.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:48.757 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:48.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:48.757 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:48.531+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:48.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:48.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:48.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:48.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:48.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:48.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:48.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:48.391+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:49.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:48.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:48.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:48.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:49.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:48.754+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:49.558 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:49.289+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18240 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:49.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:49.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:49.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:49.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:49.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:49.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:49.559 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:49.413+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:49.757+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:49.757+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:49.757+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:49.758+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:49.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:49.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:49.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:49.557+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:50.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:50.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:50.768 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:50.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:50.768 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:50.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:50.768 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:50.576+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:50.768 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:50.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:50.768 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:50.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:50.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:50.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:50.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:50.436+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:51.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:50.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:50.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:50.765+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:50.765+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:51.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:51.614+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:51.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:51.614+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:51.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:51.614+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:51.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:51.615+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:51.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:51.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:51.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:51.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:51.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:51.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:51.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:51.410+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:52.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:51.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:51.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:51.799+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:51.800+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:52.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:52.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:52.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:52.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:52.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:52.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:52.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:52.365+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:52.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:52.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:52.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:52.643+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:53.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:52.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:52.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:52.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:52.787+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:53.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:53.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:53.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:53.595+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:53.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:53.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:53.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:53.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:53.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:53.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:53.395+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:54.293 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:53.822+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:54.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:53.822+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:54.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:53.822+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:54.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:53.822+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:54.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:54.291+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18292 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:54.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:54.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:54.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:54.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:54.389+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:54.802+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:54.802+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:54.802+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:54.803+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:54.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:55.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:54.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:55.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:54.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:55.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:54.616+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:55.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:55.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:55.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:55.423+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:55.843 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:55.576+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:55.843 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:55.576+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:55.843 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:55.576+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:55.843 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:55.577+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:55.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:55.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:55.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:56.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:55.841+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:56.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:56.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:56.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:56.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:56.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:56.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:56.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:56.385+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:56.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:56.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:56.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:56.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:56.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:56.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:56.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:56.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:56.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:56.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:56.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:56.797+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:57.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:57.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:57.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:57.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:57.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:57.386+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:57.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:57.387+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:57.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:57.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:57.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:57.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:57.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:57.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:57.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:57.611+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:57.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:57.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:57.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:57.788+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:58.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:58.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:58.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:55:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:58.652+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:58.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:58.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:58.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:58.429+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:55:59.295 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:58.810+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:59.295 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:58.810+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:59.295 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:58.810+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:55:59.296 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:58.810+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:55:59.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:55:59.293+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18345 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:55:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:59.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:59.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:59.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:55:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:55:59.479+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:00.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:59.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:59.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:59.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:55:59.619+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:59.853+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:59.853+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:59.853+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:55:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:55:59.854+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:00.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:00.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:00.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:00.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:00.498+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:00.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:00.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:00.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:00.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:00.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:00.575+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:01.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:00.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:00.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:00.872+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:00.873+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:01.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:01.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:01.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:01.487+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:01.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:01.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:01.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:01.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:01.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:01.556+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:01.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:01.556+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:02.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:01.869+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:01.869+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:01.869+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:01.870+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:02.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:02.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:02.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:02.607+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:02.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:02.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:02.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:02.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:03.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:02.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:02.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:02.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:02.867+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:03.919 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:03.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:03.919 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:03.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:03.920 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:03.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:03.920 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:03.631+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:03.920 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:03.559+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:03.920 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:03.559+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:03.921 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:03.559+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:03.921 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:03.559+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:04.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:03.917+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:04.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:03.917+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:04.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:03.917+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:04.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:03.918+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:04.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:04.294+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18397 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:04.924 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:04.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:04.924 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:04.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:04.924 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:04.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:04.924 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:04.591+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:04.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:04.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:04.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:04.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:04.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:04.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:04.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:04.602+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:05.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:04.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:05.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:04.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:05.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:04.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:05.266 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:04.922+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:05.926 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:05.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:05.927 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:05.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:05.927 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:05.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:05.927 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:05.604+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:05.928 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:05.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:05.928 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:05.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:05.929 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:05.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:05.929 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:05.595+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:06.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:05.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:05.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:05.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:05.921+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:06.903 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:06.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:06.903 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:06.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:06.904 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:06.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:06.904 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:06.649+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:06.904 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:06.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:06.904 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:06.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:06.904 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:06.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:06.904 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:06.616+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:07.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:06.900+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:06.900+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:06.900+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:06.900+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:07.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:07.574+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:07.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:07.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:07.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:07.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:07.886 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:07.575+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:07.886 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:07.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:07.886 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:07.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:07.886 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:07.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:07.886 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:07.628+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:08.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:07.882+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:07.882+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:07.882+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:07.883+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:08.928 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:08.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:08.928 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:08.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:08.929 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:08.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:08.929 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:08.617+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:08.929 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:08.636+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:08.929 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:08.636+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:08.929 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:08.636+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:08.929 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:08.637+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:09.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:08.926+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:09.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:08.926+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:09.299 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:08.926+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:09.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:08.927+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:09.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:09.296+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18450 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:09.924 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:09.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:09.924 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:09.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:09.925 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:09.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:09.925 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:09.611+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:09.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:09.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:09.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:09.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:09.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:09.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:09.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:09.671+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:10.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:09.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:10.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:09.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:10.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:09.921+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:10.308 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:09.923+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:10.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:10.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:10.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:10.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:10.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:10.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:10.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:10.656+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:10.959 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:10.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:10.960 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:10.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:10.960 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:10.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:10.960 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:10.607+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:10.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:10.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:10.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:10.957+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:11.961 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:11.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:11.961 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:11.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:11.961 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:11.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:11.962 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:11.619+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:11.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:11.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:11.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:11.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:11.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:11.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:11.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:11.637+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:12.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:11.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:11.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:11.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:11.959+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:12.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:12.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:12.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:12.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:12.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:12.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:12.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:12.624+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:12.945 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:12.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:12.945 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:12.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:12.945 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:12.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:12.945 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:12.575+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:13.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:12.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:12.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:12.941+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:12.942+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:13.939 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:13.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:13.939 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:13.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:13.939 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:13.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:13.940 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:13.647+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:13.940 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:13.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:13.940 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:13.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:13.940 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:13.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:13.940 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:13.566+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:14.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:13.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:14.301 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:13.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:14.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:13.936+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:14.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:13.937+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:14.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:14.298+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18502 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:14.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:14.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:14.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:14.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:14.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:14.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:14.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:14.619+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:14.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:14.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:14.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:14.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:14.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:14.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:14.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:14.601+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:15.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:14.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:14.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:14.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:14.974+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:15.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:15.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:15.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:15.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:15.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:15.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:15.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:15.577+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:16.022 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:15.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:16.023 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:15.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:16.023 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:15.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:16.023 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:15.654+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:16.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:16.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:16.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:16.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:16.021+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:16.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:16.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:16.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:16.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:16.662+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:16.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:16.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:16.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:16.536+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.004+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.005+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:17.959 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:17.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:17.959 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:17.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:17.959 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:17.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:17.959 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:17.676+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:17.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:17.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:17.960 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:17.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:17.960 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:17.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:17.960 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:17.585+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:18.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.956+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:17.957+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:18.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:18.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:18.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:18.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:18.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:18.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:18.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:18.619+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:18.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:18.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:18.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:18.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:18.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:18.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:18.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:18.655+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:19.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:18.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:19.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:18.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:19.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:18.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:19.302 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:18.988+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:19.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:19.299+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18555 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:19.982 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:19.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:19.982 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:19.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:19.983 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:19.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:19.983 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:19.618+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:19.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:19.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:19.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:19.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:19.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:19.604+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:19.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:19.605+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:19.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:19.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:19.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:19.980+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:20.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:20.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:20.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:20.611+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:20.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:20.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:20.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:20.571+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:20.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:20.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:20.988+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:20.989+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:21.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:21.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:21.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:21.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:21.559+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:21.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:21.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:21.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:21.581+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:21.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:21.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:21.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:21.959+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:22.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:22.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:22.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:22.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:22.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:22.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:22.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:22.570+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:22.816 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:22.554+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:22.816 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:22.554+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:22.816 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:22.554+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:22.817 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:22.554+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:23.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:22.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:22.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:22.973+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:22.974+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:23.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:23.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:23.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:23.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:23.552+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:23.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:23.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:23.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:23.537+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:24.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:24.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:24.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:24.304 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.023+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:24.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:24.301+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18607 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:24.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:24.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:24.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:24.542+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:24.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:24.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:24.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:24.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:24.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:24.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:24.994 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:24.576+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:25.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:24.992+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:25.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:25.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:25.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:25.520+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:25.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:25.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:25.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:25.532+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:26.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:26.000+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:26.000+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:26.000+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:26.001+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:26.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:26.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:26.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:26.528+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:26.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:26.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:26.515+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:26.516+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:27.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.012+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:27.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.013+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:27.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:27.503+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:27.503+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:27.503+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:27.504+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:27.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:27.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:27.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:27.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:27.492+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:28.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.994+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:27.995+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:28.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:28.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:28.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:28.508+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:28.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:28.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:28.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:28.491+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:29.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:29.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:29.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:29.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:29.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:29.022+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:29.307 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:29.023+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:29.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:29.303+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18660 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:29.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:29.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:29.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:29.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:29.552+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:29.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:29.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:29.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:29.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:29.462+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:30.466 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:30.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:30.467 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:30.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:30.467 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:30.069+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:30.467 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:30.070+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:30.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:30.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:30.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:30.554+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:30.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:30.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:30.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:30.497+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:31.513 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:31.087+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:31.513 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:31.088+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:31.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:31.088+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:31.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:31.088+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:31.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:31.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:31.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:31.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:31.512+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:31.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:31.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:31.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:31.539+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:32.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:32.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:32.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:32.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:32.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:32.112+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:32.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:32.113+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:32.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:32.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:32.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:32.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:32.545+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:32.578+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:32.578+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:32.578+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:32.579+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:33.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:33.136+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:33.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:33.136+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:33.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:33.136+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:33.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:33.137+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:33.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:33.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:33.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:33.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:33.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:33.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:33.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:33.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:33.599+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:34.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:34.305+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18712 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:34.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:34.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:34.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:34.187+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:34.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:34.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:34.572+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:01.087009+0000 (oldest deadline 2024-03-12T03:46:21.087009+0000) 2024-03-12T03:56:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:34.572+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:34.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:35.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:34.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:35.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:34.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:35.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:34.613+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:35.506 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:35.197+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:35.506 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:35.197+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:35.506 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:35.197+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:35.506 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:35.198+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:35.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:35.621+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:35.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:35.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:35.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:35.645+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:36.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:36.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:36.210+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:36.211+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:36.666+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:37.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:36.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:37.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:36.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:37.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:36.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:37.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:36.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:37.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:37.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:37.236+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:37.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:37.237+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:37.642+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:37.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:37.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:37.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:37.604+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:38.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:38.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:38.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:38.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:38.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:38.639+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:38.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:38.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:38.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:38.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:38.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:38.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:38.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:38.632+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:39.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:39.306+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18765 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:39.248+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:39.248+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:39.248+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:39.249+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:40.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:39.641+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:39.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:39.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:39.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:39.616+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:40.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:40.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:40.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:40.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:40.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:40.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:40.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:40.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:40.638+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:40.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:40.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:40.615+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:40.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:40.616+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:41.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:41.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:41.299+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:41.299+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:41.651+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:41.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:41.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:41.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:41.601+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:42.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:42.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:42.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:42.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:42.273+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:42.650+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:42.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:43.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:42.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:43.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:42.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:43.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:42.645+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:43.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:43.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:43.281+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:43.282+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:44.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:43.638+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:43.691+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:43.691+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:43.691+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:43.692+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:44.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:44.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:44.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:44.303+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:44.604 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:44.308+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18817 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:44.602+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:44.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:44.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:44.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:44.735+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:45.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:45.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:45.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:45.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:45.274+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:45.567+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:45.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:45.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:45.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:10.408626+0000 (oldest deadline 2024-03-12T03:46:30.408626+0000) 2024-03-12T03:56:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:45.723+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:46.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:46.558+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:46.752 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:46.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:46.752 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:46.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:46.752 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:46.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:46:12.040235+0000 (oldest deadline 2024-03-12T03:46:32.040235+0000) 2024-03-12T03:56:46.752 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:46.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:47.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:46.749+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:47.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:47.516+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:47.292+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:48.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:47.757+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:48.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:48.260+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:48.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:48.723+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:48.559+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:49.568 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:49.310+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18870 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:49.569 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:49.309+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:50.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:49.677+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:49.566+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:50.588 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:50.332+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:51.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:50.725+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:50.586+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:51.618 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:51.363+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:52.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:51.766+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:51.616+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:52.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:52.398+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:52.746 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:52.616+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:53.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:52.741+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:53.628 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:53.349+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:54.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:53.625+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:53.708+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:54.675 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:54.311+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18921 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:54.675 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:54.373+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:55.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:54.757+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:54.672+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:55.762 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:55.423+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:55.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:55.637+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:56.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:55.760+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:56.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:56.450+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:56.739 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:56.653+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:57.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:56.737+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:57.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:57.491+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:57.681+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:57.704+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:58.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:58.500+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:58.691+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:56:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:58.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:56:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:58.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:56:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:58.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:56:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:58.641+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:56:59.663 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:56:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:56:59.314+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 18973 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:56:59.663 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:56:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:56:59.474+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:00.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:56:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:56:59.686+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:59.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:59.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:59.660+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:56:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:56:59.661+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:00.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:00.675+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:00.466+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:00.679+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:00.679+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:00.679+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:00.679+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:01.731 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:01.469+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:01.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:01.715+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:02.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:01.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:01.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:01.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:01.729+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:02.734 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:02.452+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:03.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:02.731+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:02.776+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:02.776+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:02.776+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:02.777+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:03.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:03.493+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:03.698+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:03.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:03.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:03.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:03.741+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:04.713 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:04.315+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19026 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:04.713 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:04.478+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:05.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:04.745+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:04.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:04.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:04.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:04.711+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:05.734 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:05.453+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:05.749+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:05.739+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:05.739+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:05.739+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:05.740+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:06.708 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:06.424+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:07.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:06.706+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:06.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:06.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:06.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:06.762+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:07.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:07.426+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:07.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:07.426+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:07.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:07.426+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:07.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:07.428+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:08.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:07.754+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:07.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:07.738+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:07.738+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:07.739+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:08.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:08.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:08.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:08.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:08.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:08.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:08.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:08.438+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:09.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:08.738+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:08.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:08.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:08.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:09.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:08.738+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:09.317+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19078 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:09.733 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:09.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:09.733 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:09.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:09.733 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:09.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:09.733 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:09.472+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:10.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:09.752+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:09.730+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:09.730+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:09.730+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:09.731+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:10.721 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:10.457+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:10.721 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:10.457+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:10.722 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:10.457+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:10.722 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:10.458+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:11.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:10.750+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:10.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:10.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:10.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:10.720+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:11.762 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:11.737+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:11.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:11.496+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:11.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:11.496+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:11.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:11.496+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:11.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:11.496+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:12.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:11.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:11.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:11.760+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:11.761+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:12.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:12.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:12.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:12.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:12.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:12.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:12.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:12.523+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:12.805 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:12.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:12.806 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:12.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:12.806 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:12.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:12.806 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:12.742+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:13.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:12.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:12.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:12.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:12.803+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:13.769 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:13.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:13.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:13.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:13.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:13.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:13.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:13.483+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:13.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:13.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:13.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:13.768+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:13.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:13.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:13.833+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:13.834+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:14.805 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:14.319+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19131 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:14.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:14.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:14.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:14.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:14.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:14.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:14.806 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:14.498+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:14.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:14.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:14.802+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:14.803+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:14.866+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:15.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:14.866+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:15.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:14.866+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:15.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:14.867+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:15.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:15.543+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:15.543+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:15.543+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:15.544+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:16.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:15.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:15.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:15.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:15.917+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:15.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:16.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:15.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:16.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:15.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:16.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:15.838+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:16.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:16.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:16.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:16.505+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:16.505+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:16.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:16.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:16.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:17.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:16.813+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:16.904+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:16.904+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:16.904+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:17.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:16.905+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:17.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:17.463+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:17.463+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:17.463+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:17.464+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:18.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:17.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:17.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:17.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:18.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:17.831+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:17.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:17.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:17.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:17.949+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:18.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:18.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:18.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:18.416+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:18.417+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:18.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:18.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:18.810+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:18.811+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:18.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:18.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:18.969+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:18.969+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:19.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:19.321+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19183 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:19.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:19.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:19.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:19.408+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:20.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:19.935+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:19.935+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:19.935+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:19.936+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:19.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:20.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:19.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:19.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:19.845+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:20.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:20.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:20.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:20.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:20.383+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:21.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:20.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:20.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:20.825+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:21.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:20.826+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:20.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:20.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:20.892+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:21.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:20.893+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:21.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:21.392+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:21.392+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:21.392+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:21.393+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:22.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:21.918+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:21.918+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:21.918+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:21.919+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:21.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:22.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:21.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:21.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:22.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:21.824+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:22.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:22.405+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:22.405+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:22.405+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:22.406+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:22.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:22.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:22.854+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:23.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:22.854+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:22.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:22.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:22.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:23.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:22.907+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:23.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:23.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:23.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:23.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:23.417+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:24.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:23.941+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:23.941+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:23.941+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:24.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:23.942+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:23.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:23.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:23.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:24.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:23.883+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:24.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:24.322+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19236 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:24.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:24.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:24.411+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:24.412+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:25.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:24.987+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:24.987+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:24.987+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:24.988+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:24.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:24.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:24.834+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:25.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:24.835+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:25.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:25.364+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:25.364+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:25.364+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:25.365+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:26.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:25.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:25.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:25.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:25.825+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:25.947+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:26.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:25.947+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:26.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:25.947+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:26.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:25.948+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:26.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:26.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:26.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:26.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:26.342+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:27.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:26.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:26.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:26.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:26.847+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:26.914+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:26.914+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:26.914+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:26.915+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:27.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:27.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:27.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:27.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:27.328+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:28.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:27.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:27.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:27.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:27.930+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:27.893+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:27.893+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:28.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:27.893+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:28.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:27.894+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:28.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:28.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:28.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:28.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:28.341+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:29.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:28.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:29.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:28.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:29.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:28.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:29.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:28.858+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:29.303 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:28.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:29.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:28.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:29.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:28.931+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:29.304 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:28.932+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:29.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:29.324+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19288 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:29.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:29.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:29.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:29.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:29.301+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:30.267 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:29.960+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:30.268 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:29.961+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:30.268 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:29.961+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:30.268 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:29.962+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:30.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:29.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:30.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:29.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:30.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:29.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:30.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:29.816+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:30.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:30.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:30.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:30.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:30.266+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:31.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:30.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:31.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:30.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:31.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:30.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:31.253 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:30.845+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:31.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:30.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:31.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:30.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:31.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:30.922+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:31.254 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:30.923+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:31.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:31.250+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:31.250+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:31.250+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:31.251+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:32.268 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:31.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:32.268 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:31.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:32.268 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:31.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:32.268 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:31.935+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:32.268 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:31.867+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:32.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:31.867+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:32.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:31.867+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:32.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:31.868+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:32.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:32.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:32.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:32.265+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:32.266+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:33.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:32.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:33.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:32.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:33.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:32.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:33.224 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:32.933+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:33.224 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:32.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:33.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:32.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:33.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:32.883+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:33.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:32.883+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:33.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:33.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:33.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:33.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:33.222+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:33.966 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:33.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:33.966 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:33.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:33.967 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:33.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:33.967 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:33.862+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:34.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:33.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:33.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:33.964+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:33.965+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:34.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:34.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:34.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:34.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:34.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:34.207+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:34.326+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19341 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:35.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:34.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:35.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:34.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:35.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:34.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:35.168 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:34.981+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:35.169 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:34.843+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:35.169 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:34.843+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:35.169 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:34.843+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:35.169 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:34.844+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:35.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:35.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:35.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:35.165+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:35.167+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:36.125 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:35.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:36.125 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:35.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:36.125 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:35.851+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:36.125 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:35.852+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:36.125 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:35.986+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:36.126 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:35.986+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:36.126 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:35.986+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:36.126 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:35.987+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:36.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:36.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:36.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:36.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:36.123+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:36.993 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:36.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:36.993 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:36.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:36.993 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:36.813+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:36.993 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:36.814+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:37.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:36.990+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:37.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:36.990+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:37.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:36.990+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:37.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:36.991+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:37.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:37.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:37.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:37.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:37.117+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:37.785+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:37.785+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:37.785+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:37.786+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:38.018+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:38.018+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:38.018+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:38.018+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:38.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:38.098+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:38.098+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:38.098+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:38.099+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:39.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:39.024+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:39.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:39.024+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:39.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:39.024+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:39.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:39.025+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:39.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:38.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:39.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:38.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:39.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:38.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:39.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:38.793+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:39.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:39.328+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19393 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:39.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:39.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:39.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:39.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:39.119+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:40.137 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:39.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:40.138 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:39.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:40.138 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:39.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:40.138 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:39.826+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:40.138 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:40.056+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:40.138 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:40.056+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:40.139 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:40.056+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:40.139 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:40.057+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:40.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:40.135+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:40.135+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:40.135+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:40.136+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:41.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:40.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:40.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:40.786+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:40.787+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:41.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:41.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:41.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:41.182+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:41.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:41.071+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:41.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:41.071+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:41.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:41.071+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:41.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:41.072+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:42.117 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:41.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:42.117 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:41.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:42.117 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:41.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:42.117 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:41.823+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:42.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:42.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:42.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:42.114+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:42.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:42.114+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:42.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:42.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:42.160+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:42.160+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:43.134 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:42.849+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:43.134 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:42.849+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:43.134 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:42.849+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:43.135 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:42.850+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:43.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:43.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:43.188+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:43.188+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:43.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:43.131+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:43.131+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:43.131+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:43.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:43.133+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:44.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:43.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:43.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:43.803+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:43.804+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:44.332 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:44.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:44.332 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:44.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:44.332 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:44.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:44.333 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:44.179+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:44.333 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:44.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:44.333 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:44.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:44.333 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:44.154+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:44.333 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:44.155+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:44.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:44.330+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19446 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:45.160 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:44.836+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:45.160 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:44.836+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:45.160 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:44.836+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:45.160 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:44.837+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:45.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:45.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:45.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:45.158+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:45.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:45.158+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:45.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:45.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:45.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:45.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:45.207+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:46.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:45.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:45.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:45.805+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:45.806+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:46.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:46.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:46.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:46.194+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:46.195+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:46.134+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:46.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:46.134+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:46.134+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:46.135+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:47.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:46.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:46.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:46.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:46.771+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:47.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:47.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:47.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:47.181+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:47.181+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:47.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:47.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:47.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:47.184+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:48.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:47.762+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:47.762+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:47.762+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:47.762+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:48.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:48.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:48.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:48.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:48.179+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:48.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:48.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:48.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:48.223+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:48.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:48.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:48.793+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:48.794+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:49.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:49.331+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19498 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:49.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:49.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:49.203+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:49.203+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:49.176+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:49.176+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:49.176+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:49.177+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:50.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:49.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:49.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:49.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:49.768+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:50.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:50.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:50.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:50.178+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:50.179+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:50.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:50.209+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:50.210+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:50.210+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:50.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:50.210+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:51.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:50.801+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:50.801+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:50.801+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:50.802+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:51.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:51.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:51.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:51.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:51.148+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:51.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:51.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:51.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:51.242+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:51.243+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:52.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:51.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:51.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:51.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:51.801+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:52.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:52.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:52.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:52.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:52.159+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:52.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:52.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:52.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:52.214+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:52.215+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:52.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:52.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:52.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:52.789+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:53.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:53.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:53.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:53.121+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:53.177+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:53.177+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:53.177+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:53.178+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:54.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:53.787+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:53.787+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:53.787+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:53.788+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:54.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:54.333+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19551 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:54.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:54.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:54.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:54.159+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:54.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:54.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:54.193+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:54.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:54.194+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:55.129 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:54.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:55.130 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:54.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:55.130 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:54.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:55.130 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:54.810+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:55.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:55.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:55.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:55.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:55.185+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:55.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:55.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:55.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:55.224+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:55.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:55.225+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:56.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:55.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:55.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:55.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:55.788+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:56.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:56.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:56.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:56.217+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:56.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:56.219+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:56.219+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:56.219+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:56.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:56.220+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:57.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:56.744+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:56.744+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:56.744+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:56.745+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:57.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:57.242+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:57.242+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:57.242+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:57.242+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:57.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:57.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:57.223+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:57.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:57.223+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:58.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:57.708+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:57.708+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:57.708+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:57.709+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:58.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:58.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:58.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:58.222+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:58.234+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:58.234+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:58.234+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:58.234+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:57:58.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:58.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:58.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:58.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:58.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:58.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:57:58.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:58.679+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:59.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:57:59.335+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19603 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:57:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:59.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:59.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:59.272+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:57:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:57:59.273+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:57:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:59.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:59.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:59.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:57:59.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:57:59.269+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:00.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:59.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:59.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:59.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:57:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:57:59.643+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:00.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:00.292+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:00.292+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:00.292+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:00.293+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:00.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:00.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:00.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:00.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:00.264+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:01.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:00.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:00.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:00.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:00.674+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:01.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:01.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:01.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:01.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:01.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:01.332+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:01.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:01.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:01.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:01.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:01.310+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:02.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:01.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:01.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:01.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:01.637+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:02.688 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:02.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:02.688 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:02.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:02.688 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:02.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:02.689 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:02.323+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:02.689 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:02.308+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:02.689 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:02.309+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:02.689 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:02.309+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:02.689 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:02.310+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:03.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:02.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:02.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:02.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:02.687+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:03.642 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:03.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:03.642 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:03.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:03.642 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:03.328+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:03.642 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:03.329+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:03.643 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:03.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:03.643 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:03.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:03.643 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:03.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:03.643 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:03.371+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:04.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:03.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:03.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:03.736+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:03.737+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:04.697 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:04.337+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19656 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:04.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:04.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:04.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:04.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:04.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:04.376+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:04.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:04.376+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:04.699 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:04.347+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:04.699 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:04.347+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:04.699 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:04.347+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:04.699 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:04.347+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:05.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:04.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:04.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:04.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:04.696+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:05.652 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:05.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:05.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:05.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:05.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:05.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:05.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:05.390+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:05.653 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:05.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:05.653 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:05.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:05.653 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:05.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:05.654 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:05.376+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:06.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:05.650+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:05.650+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:05.650+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:05.651+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:06.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:06.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:06.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:06.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:06.373+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:06.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:06.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:06.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:06.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:06.383+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:06.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:06.384+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:07.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:06.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:06.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:06.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:06.653+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:07.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:07.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:07.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:07.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:07.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:07.351+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:07.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:07.351+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:07.683 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:07.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:07.683 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:07.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:07.683 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:07.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:07.683 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:07.389+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:08.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:07.679+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:08.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:07.679+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:08.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:07.679+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:08.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:07.680+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:08.663 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:08.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:08.663 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:08.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:08.663 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:08.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:08.663 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:08.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:08.663 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:08.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:08.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:08.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:08.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:08.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:08.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:08.423+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:08.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:08.660+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:08.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:08.660+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:08.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:08.660+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:08.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:08.661+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:09.691 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:09.339+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19708 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:09.691 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:09.393+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:09.691 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:09.393+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:09.691 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:09.393+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:09.692 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:09.393+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:09.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:09.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:09.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:09.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:09.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:09.408+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:09.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:09.409+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:10.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:09.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:09.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:09.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:09.688+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:10.654 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:10.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:10.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:10.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:10.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:10.370+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:10.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:10.371+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:10.655 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:10.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:10.655 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:10.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:10.656 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:10.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:10.656 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:10.365+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:11.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:10.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:10.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:10.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:10.653+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:11.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:11.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:11.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:11.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:11.398+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:11.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:11.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:11.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:11.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:11.323+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:11.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:11.324+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:12.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:11.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:11.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:11.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:11.683+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:12.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:12.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:12.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:12.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:12.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:12.367+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:12.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:12.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:12.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:12.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:12.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:12.285+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:13.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:12.719+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:12.719+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:12.719+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:12.720+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:13.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:13.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:13.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:13.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:13.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:13.364+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:13.291+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:13.291+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:13.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:13.291+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:13.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:13.293+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:14.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:13.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:13.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:13.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:13.693+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:14.696 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:14.341+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19761 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:14.696 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:14.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:14.696 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:14.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:14.697 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:14.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:14.697 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:14.325+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:14.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:14.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:14.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:14.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:14.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:14.405+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:14.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:14.406+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:15.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:14.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:14.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:14.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:14.694+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:15.656 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:15.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:15.657 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:15.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:15.657 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:15.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:15.657 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:15.342+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:15.657 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:15.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:15.657 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:15.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:15.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:15.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:15.658 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:15.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:15.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:15.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:15.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:15.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:15.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:15.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:16.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:15.655+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:16.633 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:16.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:16.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:16.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:16.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:16.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:16.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:16.368+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:16.634 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:16.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:16.634 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:16.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:16.635 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:16.374+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:16.635 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:16.375+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:17.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:16.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:16.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:16.661+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:16.663+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:17.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:17.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:17.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:17.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:17.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:17.360+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:17.655 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:17.361+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:17.656 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:17.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:17.656 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:17.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:17.656 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:17.368+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:17.656 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:17.369+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:18.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:17.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:17.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:17.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:17.653+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:18.697 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:18.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:18.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:18.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:18.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:18.369+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:18.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:18.369+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:18.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:18.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:18.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:18.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:18.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:18.327+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:18.699 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:18.328+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:18.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:18.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:18.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:18.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:18.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:18.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:18.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:18.696+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:19.686 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:19.343+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19813 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:19.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:19.356+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:19.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:19.356+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:19.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:19.356+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:19.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:19.357+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:19.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:19.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:19.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:19.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:19.688 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:19.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:19.688 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:19.373+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:20.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:19.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:20.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:19.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:20.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:19.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:20.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:19.685+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:20.704 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:20.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:20.704 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:20.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:20.704 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:20.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:20.704 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:20.378+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:20.705 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:20.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:20.705 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:20.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:20.705 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:20.414+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:20.705 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:20.415+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:21.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:20.701+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:20.701+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:20.701+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:20.702+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:21.680 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:21.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:21.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:21.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:21.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:21.415+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:21.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:21.416+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:21.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:21.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:21.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:21.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:21.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:21.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:21.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:21.402+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:21.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:21.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:21.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:21.679+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:22.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:22.413+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:22.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:22.413+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:22.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:22.413+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:22.705 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:22.414+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:22.706 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:22.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:22.706 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:22.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:22.706 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:22.447+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:22.706 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:22.447+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:23.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:22.702+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:22.702+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:22.702+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:22.703+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:23.694 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:23.413+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:23.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:23.413+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:23.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:23.413+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:23.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:23.414+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:23.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:23.483+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:23.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:23.483+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:23.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:23.483+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:23.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:23.484+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:24.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:23.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:23.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:23.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:23.693+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:24.666 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:24.345+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19866 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:24.666 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:24.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:24.666 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:24.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:24.666 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:24.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:24.667 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:24.503+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:24.667 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:24.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:24.667 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:24.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:24.667 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:24.461+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:24.667 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:24.461+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:25.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:24.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:24.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:24.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:24.664+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:25.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:25.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:25.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:25.459+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:25.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:25.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:25.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:25.673+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:25.479+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:25.479+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:25.479+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:25.479+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:26.711 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:26.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:26.711 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:26.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:26.711 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:26.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:26.712 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:26.470+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:26.712 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:26.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:26.712 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:26.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:26.712 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:26.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:26.712 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:26.498+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:27.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:26.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:26.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:26.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:26.710+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:27.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:27.481+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:27.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:27.481+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:27.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:27.481+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:27.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:27.482+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:27.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:27.434+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:27.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:27.434+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:27.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:27.434+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:27.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:27.435+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:28.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:27.743+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:27.743+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:27.743+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:27.743+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:28.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:28.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:28.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:28.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:28.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:28.400+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:28.757 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:28.400+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:28.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:28.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:28.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:28.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:28.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:28.484+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:28.758 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:28.484+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:29.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:28.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:29.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:28.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:29.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:28.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:29.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:28.754+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:29.803 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:29.346+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19918 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:29.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:29.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:29.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:29.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:29.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:29.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:29.803 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:29.463+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:29.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:29.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:29.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:29.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:29.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:29.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:29.804 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:29.423+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:30.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:29.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:30.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:29.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:29.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:29.801+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:30.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:30.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:30.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:30.482+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:30.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:30.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:30.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:30.447+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:31.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:30.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:30.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:30.824+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:30.825+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:31.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:31.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:31.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:31.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:31.459+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:31.487+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:31.487+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:31.487+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:31.488+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:32.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:31.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:31.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:31.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:31.817+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:32.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:32.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:32.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:32.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:32.489+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:32.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:32.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:32.464+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:32.465+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:33.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:32.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:32.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:32.837+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:32.838+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:33.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:33.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:33.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:33.426+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:33.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:33.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:33.493+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:33.494+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:33.863+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:33.863+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:33.863+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:34.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:33.863+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:34.349+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 19971 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:34.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:34.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:34.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:34.527+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:34.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:34.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:34.386+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:34.387+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:35.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:34.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:34.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:34.838+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:34.839+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:35.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:35.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:35.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:35.424+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:35.425+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:35.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:35.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:35.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:35.567+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:36.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:35.826+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:35.826+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:35.826+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:35.827+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:36.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:36.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:36.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:36.397+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:36.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:36.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:36.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:36.523+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:37.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:36.859+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:36.859+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:36.859+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:37.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:36.859+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:37.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:37.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:37.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:37.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:37.548+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:37.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:37.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:37.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:37.378+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:38.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:37.904+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:38.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:38.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:38.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:38.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:38.575+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:38.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:38.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:38.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:38.345+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:39.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:38.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:38.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:38.868+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:38.869+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:39.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:39.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:39.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:39.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:39.342+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:39.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:39.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:39.531+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:39.531+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:39.812 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:39.350+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20020 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:40.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:39.896+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:39.896+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:39.896+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:39.897+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:40.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:40.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:40.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:40.525+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:40.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:40.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:40.335+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:40.335+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:41.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:40.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:40.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:40.925+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:40.925+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:41.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:41.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:41.338+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:41.339+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:41.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:41.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:41.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:41.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:42.295 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:41.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:42.295 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:41.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:42.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:41.955+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:42.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:41.956+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:42.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:42.292+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:42.292+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:42.292+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:42.293+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:42.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:42.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:42.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:42.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:42.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:42.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:42.965 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:42.571+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:43.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:42.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:43.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:42.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:43.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:42.962+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:43.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:42.963+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:43.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:43.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:43.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:43.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:43.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:43.263+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:43.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:43.263+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:43.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:43.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:43.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:43.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:43.527+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:44.256 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:43.994+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:44.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:43.994+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:44.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:43.994+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:44.257 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:43.995+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:44.538 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:44.352+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20070 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:44.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:44.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:44.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:44.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:44.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:44.254+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:44.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:44.255+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:44.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:44.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:44.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:44.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:44.537+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:45.269 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:45.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:45.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.016+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:45.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.017+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:45.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:45.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:45.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:45.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:45.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:45.267+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:45.543 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:45.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:45.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:45.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:45.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:45.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:45.541+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:46.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:46.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:46.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.983+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:46.273 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:45.983+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:46.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:46.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:46.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:46.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:46.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:46.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:46.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:46.271+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:46.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:46.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:46.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:46.585+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:47.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:47.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:47.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:47.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:47.011+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:47.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:47.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:47.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:47.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:47.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:47.313+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:47.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:47.314+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:48.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:47.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:48.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:47.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:48.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:47.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:48.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:47.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:48.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:48.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:48.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:48.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:48.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:48.013+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:48.303 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:48.014+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:48.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:48.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:48.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:48.300+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:48.301+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:48.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:48.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:48.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:48.601+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:49.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:49.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:49.270 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:49.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:49.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:49.044+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:49.271 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:49.045+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:49.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:49.354+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20113 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:49.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:49.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:49.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:49.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:49.269+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:50.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:49.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:49.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:49.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:49.570+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:50.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:50.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:50.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:50.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:50.061+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:50.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:50.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:50.237+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:50.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:50.238+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:51.045 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:50.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:51.045 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:50.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:51.045 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:50.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:51.045 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:50.587+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:51.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:51.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:51.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:51.275+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:51.043+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:51.043+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:51.043+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:51.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:51.043+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:51.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:51.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:51.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:51.538+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:52.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:51.999+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:51.999+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:51.999+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:52.000+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:52.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:52.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:52.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:52.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:52.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:52.234+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:52.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:52.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:52.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:52.497+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:52.498+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:53.245 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:52.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:53.246 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:52.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:53.246 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:52.980+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:53.246 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:52.980+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:53.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:53.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:53.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:53.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:53.242+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:53.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:53.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:53.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:53.483+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:53.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:53.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:53.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:54.222 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:53.969+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:54.510 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:54.356+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20157 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:54.511 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:54.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:54.511 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:54.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:54.511 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:54.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:54.511 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:54.220+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:54.508+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:54.508+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:54.508+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:54.509+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:55.237 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:54.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:55.237 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:54.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:55.237 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:54.937+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:55.237 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:54.938+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:55.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:55.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:55.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:55.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:55.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:55.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:55.531 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:55.235+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:55.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:55.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:55.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:55.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:55.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:56.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:55.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:56.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:55.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:56.234 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:55.899+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:56.235 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:55.900+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:56.555 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:56.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:56.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:56.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:56.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:56.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:56.556 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:56.233+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:56.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:56.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:56.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:56.553+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:56.554+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:57.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:56.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:57.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:56.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:57.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:56.858+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:57.186 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:56.859+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:57.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:57.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:57.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:57.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:57.184+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:58.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:57.818+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:57.818+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:57.818+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:57.819+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:57.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:57.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:57.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:57.582+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:58.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:58.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:58.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:58.148+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:58.149+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:58.826 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:58.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:58.826 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:58.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:58.826 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:58.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:58.826 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:58.573+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:58:59.121 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:58.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:59.122 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:58.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:59.122 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:58.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:58:59.122 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:58.824+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:59.549 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:58:59.358+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20202 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:58:59.549 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:59.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:59.549 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:59.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:59.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:59.119+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:58:59.550 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:58:59.120+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:58:59.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:59.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:59.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:59.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:58:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:58:59.547+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:59.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:59.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:59.819+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:00.123 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:58:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:58:59.820+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:00.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:00.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:00.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:00.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:00.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:00.120+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:00.514 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:00.121+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:00.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:00.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:00.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:00.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:00.512+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:01.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:00.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:00.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:00.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:01.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:00.854+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:01.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:01.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:01.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:01.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:01.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:01.083+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:01.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:01.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:01.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:01.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:01.536+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:02.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:01.873+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:01.873+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:01.873+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:02.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:01.874+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:02.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:02.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:02.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:02.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:02.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:02.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:02.034+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:02.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:02.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:02.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:02.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:02.552+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:03.024+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:03.024+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:03.024+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:03.024+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:02.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:02.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:02.909+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:03.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:02.909+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:03.941 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:03.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:03.941 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:03.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:03.942 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:03.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:03.942 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:03.586+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:04.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:03.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:03.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:03.939+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:03.939+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:04.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:04.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:04.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:04.021+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:04.618 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:04.360+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20249 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:04.971 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:04.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:04.971 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:04.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:04.971 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:04.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:04.971 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:04.617+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:05.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:05.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:05.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:05.028+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:04.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:04.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:04.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:05.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:04.969+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:05.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:05.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:05.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:05.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:05.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:05.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:05.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:05.586+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:06.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:06.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:06.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:06.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:06.078+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:05.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:05.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:05.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:06.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:05.947+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:06.902 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:06.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:06.902 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:06.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:06.903 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:06.595+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:06.903 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:06.595+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:07.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:06.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:06.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:06.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:06.901+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:07.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:07.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:07.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:07.054+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:07.902 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:07.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:07.902 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:07.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:07.902 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:07.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:07.903 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:07.559+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:08.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:07.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:07.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:07.900+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:07.901+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:08.060+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:08.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:08.061+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:08.062+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:08.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:08.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:08.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:08.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:08.539+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:09.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:09.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:09.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:09.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:09.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:09.041+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:09.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:08.902+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:09.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:08.902+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:08.902+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:09.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:08.902+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:09.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:09.362+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20298 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:09.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:09.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:09.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:09.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:09.575+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:10.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:10.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:10.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:10.071+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:10.072+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:09.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:09.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:09.887+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:10.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:09.887+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:10.855 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:10.577+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:10.855 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:10.577+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:10.856 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:10.577+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:10.856 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:10.577+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:11.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:11.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:11.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:11.027+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:11.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:11.028+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:10.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:11.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:10.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:10.853+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:11.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:10.854+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:12.036 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:11.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:12.036 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:11.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:12.036 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:11.839+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:12.037 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:11.840+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:12.037 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:11.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:12.037 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:11.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:12.037 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:11.613+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:12.037 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:11.614+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:12.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:12.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.035+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:12.988 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:12.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:12.989 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:12.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:12.989 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:12.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:12.989 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:12.816+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:12.989 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:12.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:12.989 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:12.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:12.989 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:12.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:12.990 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:12.649+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:13.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.986+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.986+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.986+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:13.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:12.987+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:13.994 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:13.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:13.994 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:13.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:13.994 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:13.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:13.994 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:13.693+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:13.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:13.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:13.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:13.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:13.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:13.770+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:13.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:13.771+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:14.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:13.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:13.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:13.991+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:14.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:13.992+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:14.732 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:14.364+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20350 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:15.001 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:14.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:15.001 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:14.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:15.002 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:14.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:15.002 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:14.810+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:15.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:14.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:15.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:14.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:15.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:14.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:15.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:14.730+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:15.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:15.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:15.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:15.040+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:15.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:15.041+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:16.025 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:15.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:16.025 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:15.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:16.025 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:15.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:16.026 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:15.715+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:16.026 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:15.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:16.026 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:15.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:16.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:15.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:16.027 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:15.790+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:16.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:16.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:16.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:16.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:16.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:16.023+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:17.056 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:16.762+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:17.056 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:16.762+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:17.056 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:16.763+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:17.057 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:16.763+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:17.057 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:16.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:17.057 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:16.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:17.057 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:16.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:17.057 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:16.759+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:17.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:17.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:17.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:17.054+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:17.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:17.054+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:18.027 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:17.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:18.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:17.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:18.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:17.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:18.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:17.753+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:18.028 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:17.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:18.028 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:17.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:18.029 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:17.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:18.029 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:17.793+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:18.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:18.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:18.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:18.025+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:18.026+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:19.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:18.797+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:19.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:18.797+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:19.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:18.797+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:19.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:18.798+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:19.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:18.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:19.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:18.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:19.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:18.740+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:19.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:18.741+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:19.368 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:19.070+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:19.368 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:19.070+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:19.369 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:19.070+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:19.369 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:19.070+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:19.768 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:19.366+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20400 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:19.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:19.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:19.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:19.766+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:20.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:19.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:20.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:19.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:20.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:19.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:20.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:19.780+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:20.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:20.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:20.109+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:20.109+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:20.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:20.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:20.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:20.738+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:21.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:20.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:21.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:20.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:21.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:20.816+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:21.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:20.818+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:21.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:21.139+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:21.139+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:21.139+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:21.140+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:22.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:21.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:21.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:21.800+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:21.801+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:21.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:21.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:22.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:21.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:22.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:21.761+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:22.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:22.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:22.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:22.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:22.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:22.126+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:22.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:22.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:22.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:23.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:22.774+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:22.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:22.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:22.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:22.730+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:23.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:23.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:23.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:23.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:23.177+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:23.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:23.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:23.761+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:23.762+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:23.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:23.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:23.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:23.726+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:24.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:24.368+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20453 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:24.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:24.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:24.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:24.216+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:25.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:24.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:24.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:24.789+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:24.789+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:24.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:24.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:24.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:24.770+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:25.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:25.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:25.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:25.233+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:25.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:25.233+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:25.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:25.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:25.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:25.759+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:25.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:25.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:25.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:25.745+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:26.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:26.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:26.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:26.184+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:26.185+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:27.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:26.726+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:26.726+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:26.726+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:26.727+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:26.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:26.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:26.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:27.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:26.800+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:27.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:27.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:27.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:27.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:27.199+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:28.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:27.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:27.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:27.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:27.832+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:27.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:27.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:27.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:28.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:27.728+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:28.213+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:28.213+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:28.213+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:28.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:28.214+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:28.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:28.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:28.827+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:28.828+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:28.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:28.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:28.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:28.684+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:29.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:29.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:29.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:29.217+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:29.217+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:29.671 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:29.370+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20505 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:30.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:29.855+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:29.855+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:29.855+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:30.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:29.856+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:29.669+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:29.669+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:30.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:29.669+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:30.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:29.669+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:30.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:30.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:30.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:30.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:30.264+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:30.896 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:30.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:30.896 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:30.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:30.896 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:30.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:30.896 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:30.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:31.254 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:30.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:31.254 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:30.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:31.255 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:30.894+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:31.255 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:30.894+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:31.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:31.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:31.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:31.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:31.253+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:31.872 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:31.591+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:31.873 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:31.591+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:31.873 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:31.591+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:31.873 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:31.592+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:32.261 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:31.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:32.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:31.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:32.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:31.870+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:32.262 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:31.871+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:32.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:32.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:32.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:32.259+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:32.260+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:32.880 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:32.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:32.880 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:32.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:32.880 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:32.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:32.880 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:32.584+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:33.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:32.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:33.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:32.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:33.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:32.878+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:33.266 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:32.879+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:33.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:33.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:33.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:33.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:33.264+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:34.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:33.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:34.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:33.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:34.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:33.845+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:34.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:33.846+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:34.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:33.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:34.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:33.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:34.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:33.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:34.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:33.628+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:34.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:34.372+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20558 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:34.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:34.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:34.222+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:34.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:34.223+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:35.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:34.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:34.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:34.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:35.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:34.865+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:35.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:34.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:34.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:34.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:34.628+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:35.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:35.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:35.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:35.181+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:35.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:35.182+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:36.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:35.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:35.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:35.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:36.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:35.656+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:35.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:35.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:35.823+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:36.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:35.824+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:36.505 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:36.166+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:36.505 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:36.166+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:36.506 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:36.166+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:36.506 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:36.168+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:36.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:36.652+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:36.652+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:36.652+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:36.653+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:37.157 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:36.867+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:37.158 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:36.867+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:37.158 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:36.867+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:37.158 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:36.867+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:37.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:37.155+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:37.155+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:37.155+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:37.156+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:37.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:37.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:37.829+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:38.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:37.829+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:37.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:37.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:37.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:38.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:37.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:38.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:38.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:38.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:38.186+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:38.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:38.186+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:38.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:38.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:38.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:38.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:38.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:38.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:38.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:38.678+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:38.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:38.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:38.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:38.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:38.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:38.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:38.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:38.871+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:39.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:39.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:39.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:39.212+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:39.212+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:39.699 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:39.373+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20610 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:39.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:39.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:39.830+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:39.831+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:39.697+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:39.697+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:39.697+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:39.698+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:40.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:40.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:40.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:40.183+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:40.184+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:40.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:40.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:40.817+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:40.818+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:40.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:40.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:40.677+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:40.678+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:41.225+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:41.225+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:41.225+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:41.226+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:41.978 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:41.707+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:41.978 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:41.707+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:41.978 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:41.707+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:41.978 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:41.708+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:41.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:41.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:41.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:41.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:41.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:41.782+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:41.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:41.783+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:42.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:42.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:42.221+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:42.222+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:42.812+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:42.812+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:42.812+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:42.813+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:42.675+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:42.675+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:42.675+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:42.676+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:43.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:43.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:43.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:43.176+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:43.177+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:43.667+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:43.667+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:43.667+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:43.668+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:43.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:43.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:43.767+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:43.768+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:44.378 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:44.167+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:44.378 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:44.167+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:44.378 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:44.167+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:44.378 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:44.168+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:44.670 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:44.375+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20663 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:45.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:44.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:44.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:44.809+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:44.810+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:44.667+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:45.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:44.667+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:45.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:44.667+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:45.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:44.668+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:45.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:45.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:45.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:45.118+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:45.119+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:46.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:45.678+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:45.678+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:45.678+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:45.679+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:45.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:45.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:46.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:45.822+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:46.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:45.823+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:46.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:46.144+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:46.144+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:46.144+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:46.144+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:47.014 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:46.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:47.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:46.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:47.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:46.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:47.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:46.815+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:47.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:46.726+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:47.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:46.726+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:47.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:46.726+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:47.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:46.726+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:47.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:47.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:47.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:47.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:47.170+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:48.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:47.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:47.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:47.846+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:47.847+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:47.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:47.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:47.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:48.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:47.684+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:48.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:48.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:48.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:48.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:48.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:48.207+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:48.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:48.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:48.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:48.715+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:48.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:48.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:48.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:48.843+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:49.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:49.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:49.215+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:49.215+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:49.702 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:49.377+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20715 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:50.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:49.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:49.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:49.840+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:49.841+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:50.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:49.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:49.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:49.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:49.701+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:50.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:50.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:50.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:50.244+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:50.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:50.245+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:51.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:50.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:50.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:50.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:50.712+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:50.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:50.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:51.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:50.815+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:51.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:50.816+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:51.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:51.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:51.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:51.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:51.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:51.217+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:52.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:51.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:52.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:51.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:51.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:51.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:51.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:51.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:51.844+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:51.844+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:52.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:52.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:52.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:52.216+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:52.217+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:52.869 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:52.755+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:52.869 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:52.755+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:52.869 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:52.755+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:52.869 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:52.756+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:53.179 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:52.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:53.179 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:52.866+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:53.180 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:52.867+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:53.180 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:52.868+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:53.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:53.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:53.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:53.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:53.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:53.178+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:54.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:53.885+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:53.885+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:53.885+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:53.886+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:54.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:53.771+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:53.771+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:53.771+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:53.772+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:54.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:54.379+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20767 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:54.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:54.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:54.198+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:54.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:54.199+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:55.155 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:54.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:55.156 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:54.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:55.156 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:54.928+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:55.156 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:54.929+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:55.156 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:54.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:55.156 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:54.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:55.157 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:54.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:55.157 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:54.814+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:55.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:55.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:55.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:55.153+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:55.154+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:56.127 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:55.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:56.127 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:55.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:56.127 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:55.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:56.127 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:55.960+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:56.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:55.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:56.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:55.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:56.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:55.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:56.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:55.811+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:56.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:56.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:56.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:56.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:56.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:56.125+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:57.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:56.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:56.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:56.970+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:56.971+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:56.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:56.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:56.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:56.808+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:57.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:57.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:57.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:57.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:57.125+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:58.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:57.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:57.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:57.948+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:57.949+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:57.776+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:57.776+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:57.776+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:57.777+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:58.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:58.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:58.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:58.121+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:58.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:58.122+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:59.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:58.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:58.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:58.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T03:59:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:58.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T03:59:59.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:58.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:59.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:58.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:59.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:58.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T03:59:59.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:58.757+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T03:59:59.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T03:59:59.381+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20820 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T03:59:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:59.168+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:59.168+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:59.168+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T03:59:59.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T03:59:59.168+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:59.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:59.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:59.773+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T03:59:59.774+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:59.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:59.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:59.908+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 03:59:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T03:59:59.909+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:00.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:00.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:00.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:00.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:00.157+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:01.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:00.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:00.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:00.918+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:00.918+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:00.747+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:00.747+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:00.747+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:00.748+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:01.128+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:01.128+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:01.128+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:01.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:01.128+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:01.926+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:01.926+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:01.926+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:01.926+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:01.701+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:01.701+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:01.701+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:01.701+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:02.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:02.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:02.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:02.156+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:02.157+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:02.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:02.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:02.951+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:02.952+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:02.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:02.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:02.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:02.711+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:03.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:03.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:03.110+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:03.111+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:03.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:03.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:03.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:03.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:03.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:03.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:03.993 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:03.745+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:03.990+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:03.990+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:03.990+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:04.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:03.991+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:04.133+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:04.133+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:04.133+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:04.134+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:04.719 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:04.383+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20872 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:04.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:04.717+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:04.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:04.717+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:04.986 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:04.717+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:04.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:04.718+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:04.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:04.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:04.984+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:05.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:04.985+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:05.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:05.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:05.096+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:05.097+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:05.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:05.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:05.959+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:05.960+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:05.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:05.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:05.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:05.730+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:06.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:06.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:06.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:06.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:06.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:06.076+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:07.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:06.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:06.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:06.927+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:06.928+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:06.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:06.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:06.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:06.766+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:07.072+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:07.072+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:07.072+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:07.074+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:07.970 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:07.784+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:07.970 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:07.784+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:07.970 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:07.784+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:07.971 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:07.785+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:08.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:07.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:07.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:07.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:07.968+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:08.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:08.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:08.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:08.030+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:08.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:08.031+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:09.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:08.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:09.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:08.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:09.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:08.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:09.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:08.788+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:09.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:08.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:09.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:08.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:09.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:08.936+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:09.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:08.937+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:09.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:09.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:09.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:09.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:09.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:09.076+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:09.388 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:09.077+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:09.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:09.385+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20925 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:10.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:09.830+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:10.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:09.830+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:10.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:09.830+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:10.092 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:09.831+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:10.092 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:09.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:10.093 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:09.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:10.093 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:09.977+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:10.093 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:09.978+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:10.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:10.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:10.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:10.089+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:10.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:10.090+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:11.127 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:10.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:11.127 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:10.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:11.127 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:10.810+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:11.128 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:10.810+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:11.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:10.945+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:11.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:10.945+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:11.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:10.946+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:11.128 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:10.946+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:11.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:11.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:11.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:11.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:11.125+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:11.125+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:11.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:11.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:11.960+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:11.960+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:11.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:11.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:11.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:11.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:12.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:12.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:12.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:12.137+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:12.138+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:13.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:12.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:12.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:12.920+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:12.921+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:12.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:12.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:12.769+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:12.770+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:13.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:13.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:13.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:13.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:13.123+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:14.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:13.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:13.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:13.897+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:13.898+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:13.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:13.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:13.814+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:13.815+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:14.390 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:14.113+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:14.390 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:14.113+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:14.390 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:14.113+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:14.390 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:14.114+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:14.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:14.387+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 20977 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:15.125 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:14.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:15.125 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:14.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:15.125 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:14.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:15.125 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:14.863+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:15.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:14.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:15.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:14.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:15.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:14.940+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:15.126 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:14.940+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:15.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:15.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:15.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:15.122+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:15.123+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:15.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:15.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:15.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:16.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:15.912+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:16.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:15.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:16.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:15.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:16.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:15.978+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:16.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:15.979+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:16.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:16.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:16.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:16.083+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:16.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:16.084+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:17.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:17.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:17.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:17.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:17.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:17.078+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:17.251 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:17.079+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:17.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:17.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:17.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:17.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:17.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:17.026+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:17.252 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:17.026+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:17.252 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:16.960+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:17.252 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:16.960+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:17.253 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:16.960+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:17.253 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:16.960+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:18.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:18.084+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:18.084+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:18.084+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:18.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:18.085+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:18.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:17.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:17.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:17.997+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:18.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:17.998+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.015+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.015+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.015+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:18.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.016+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:19.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.993+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.993+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.993+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:18.994+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:19.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:19.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:19.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:19.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:19.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:19.058+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:19.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:19.059+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:18.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:18.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:19.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:18.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:19.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:18.959+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:19.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:19.389+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21030 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:20.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:20.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:20.041+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:20.041+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:20.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.040+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:20.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:19.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:20.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:19.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:20.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:19.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:20.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:19.976+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:21.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:20.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:20.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:20.948+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:20.949+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:21.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:21.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:21.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:21.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:21.048+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:21.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:21.048+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:21.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.993+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:21.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.993+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:21.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.993+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:21.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:20.994+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:22.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:22.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:22.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:22.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:22.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:22.075+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:22.294 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:22.076+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:22.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:21.945+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:22.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:21.945+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:22.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:21.945+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:22.295 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:21.947+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:22.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:22.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:22.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:22.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:22.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:22.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:22.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:22.013+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:23.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:22.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:22.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:22.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:23.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:22.991+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:23.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:23.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:23.050+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:23.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:23.050+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:23.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:23.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:23.032+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:23.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:23.032+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:24.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:24.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:24.017+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:24.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:24.017+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:24.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:24.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.001+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:24.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.002+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:23.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:23.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:23.958+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:24.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:23.959+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:24.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:24.391+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21082 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:25.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:24.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:24.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:24.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:25.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:24.935+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:25.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:25.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:25.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:25.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:25.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:25.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:25.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:25.011+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:25.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:25.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:25.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.968+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:25.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:24.969+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:26.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:26.056+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:26.056+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:26.056+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:26.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:26.057+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:25.927+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:25.927+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:25.927+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:26.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:25.928+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:26.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:25.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:26.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:25.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:26.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:25.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:26.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:25.951+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:27.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:26.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:26.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:26.958+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:27.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:26.959+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:27.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:27.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:27.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:27.105+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:27.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:27.106+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:26.998+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:26.998+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:27.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:26.998+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:27.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:26.999+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:28.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:28.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:28.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:28.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:28.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:28.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:27.938+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:27.938+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:28.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:27.938+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:28.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:27.939+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:28.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:27.970+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:28.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:27.970+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:28.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:27.970+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:28.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:27.971+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:29.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:29.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:29.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:29.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:29.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:28.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:29.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:28.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:29.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:28.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:29.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:28.989+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:29.000+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:29.000+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:29.000+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:29.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:29.001+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:29.393+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21135 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:30.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:30.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:30.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:30.007+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:30.007+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.009+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.009+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.009+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:30.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.010+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:30.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:30.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:30.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:30.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:30.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:30.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:30.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:30.120+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:31.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:31.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:31.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:31.053+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:31.053+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:31.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:31.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:31.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:31.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:31.109+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:31.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:31.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:31.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:30.976+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:32.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:32.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:32.134+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:32.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:32.135+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:32.052+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:32.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:32.052+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:32.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:32.052+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:32.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:32.053+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:32.016+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:32.016+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:32.016+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:32.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:32.017+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:33.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:33.124+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:33.125+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:33.125+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:33.125+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:33.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:33.031+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:33.032+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:33.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:33.033+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:33.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:33.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:33.094+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:33.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:33.095+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:34.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:34.062+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:34.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:34.062+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:34.398 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:34.062+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:34.399 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:34.063+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:34.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:34.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:34.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:34.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:34.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:34.070+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:34.399 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:34.071+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:34.399 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:34.079+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:34.400 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:34.079+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:34.400 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:34.079+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:34.400 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:34.080+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:34.396+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21187 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:35.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:35.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:35.101+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:35.102+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:35.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:35.102+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:35.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:35.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:35.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:35.069+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:35.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:35.069+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:35.083+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:35.083+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:35.083+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:35.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:35.084+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:36.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:36.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:36.116+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:36.117+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:36.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:36.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:36.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:36.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:36.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:36.104+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:36.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:36.104+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:36.109+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:36.109+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:36.109+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:36.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:36.110+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:37.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:37.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:37.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:37.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:37.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:37.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:37.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:37.092+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:37.417 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:37.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:37.417 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:37.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:37.417 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:37.082+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:37.417 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:37.083+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:37.417 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:37.133+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:37.417 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:37.133+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:37.417 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:37.133+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:37.418 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:37.133+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:38.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:38.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:38.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:38.093+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:38.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:38.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:38.086+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:38.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:38.088+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:38.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:38.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:38.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:38.043+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:38.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:38.044+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:39.104+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:39.104+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:39.104+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:39.105+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:39.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:39.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:39.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:39.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:39.118+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:39.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:39.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:39.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:39.056+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:39.313 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:39.057+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:39.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:39.398+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21240 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:40.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:40.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:40.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:40.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:40.155+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:40.102+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:40.102+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:40.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:40.102+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:40.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:40.102+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.021+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:40.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.022+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:41.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:41.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:41.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:41.131+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:41.131+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:41.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:41.058+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:41.058+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:41.058+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:41.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:41.059+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.977+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.977+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.977+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:41.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:40.978+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:42.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:42.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:42.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:42.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:42.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:42.171+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:41.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:41.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:42.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:41.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:42.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:41.936+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:42.093+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:42.093+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:42.093+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:42.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:42.094+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:43.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:43.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:43.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:43.074+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:43.075+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:43.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:43.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:43.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:43.157+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:43.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:43.158+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:43.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:42.984+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:43.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:42.984+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:43.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:42.984+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:43.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:42.985+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:44.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:44.168+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:44.168+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:44.168+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:44.169+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:43.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:43.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:43.990+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:44.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:43.991+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:44.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:44.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:44.062+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:44.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:44.063+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:44.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:44.399+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21292 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:45.201+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:45.201+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:45.201+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:45.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:45.202+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:45.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:45.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:45.061+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:45.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:45.062+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:44.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:44.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:44.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:45.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:44.981+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:46.038 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:45.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:46.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:45.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:46.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:45.989+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:46.039 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:45.990+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:46.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:46.165+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:46.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:46.165+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:46.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:46.165+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:46.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:46.166+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:46.291 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.035+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:46.291 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.035+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:46.291 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.035+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:46.291 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.036+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:47.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:46.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:46.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:46.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:47.001+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:47.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:47.120+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:47.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:47.120+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:47.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:47.120+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:47.312 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:47.121+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.994+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:47.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:46.996+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:48.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:48.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:48.101+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:48.103+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:48.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:48.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:48.034+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:48.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:48.035+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:48.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:47.986+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:48.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:47.986+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:48.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:47.986+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:48.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:47.987+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:49.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:49.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:49.139+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:49.139+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:49.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:49.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:49.020+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:49.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:49.020+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:48.953+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:48.953+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:49.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:48.953+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:49.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:48.953+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:49.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:49.400+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21345 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:50.170 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:50.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:50.171 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:50.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:50.171 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:50.039+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:50.171 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:50.040+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:50.171 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:49.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:50.171 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:49.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:50.171 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:49.911+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:50.172 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:49.911+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:50.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:50.168+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:50.168+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:50.168+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:50.168+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:51.168 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:51.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:51.168 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:51.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:51.168 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:51.063+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:51.169 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:51.064+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:51.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:50.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:51.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:50.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:51.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:50.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:51.169 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:50.900+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:51.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:51.166+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:51.166+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:51.166+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:51.166+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:52.122 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:52.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:52.122 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:52.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:52.122 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:52.046+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:52.122 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:52.047+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:52.122 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:51.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:52.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:51.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:52.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:51.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:52.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:51.862+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:52.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:52.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:52.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:52.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:52.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:52.119+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:52.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:52.120+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:53.118 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:52.860+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:53.118 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:52.860+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:53.118 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:52.860+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:53.118 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:52.861+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:53.119 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:53.090+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:53.119 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:53.090+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:53.119 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:53.090+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:53.119 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:53.090+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:53.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:53.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:53.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:53.115+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:53.116+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:54.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:53.866+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:54.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:53.866+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:54.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:53.866+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:54.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:53.868+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:54.406 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:54.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:54.406 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:54.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:54.407 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:54.108+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:54.407 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:54.109+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:54.407 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:54.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:54.407 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:54.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:54.407 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:54.138+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:54.407 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:54.138+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:54.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:54.403+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21397 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:00:55.161 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:54.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:55.161 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:54.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:55.161 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:54.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:55.161 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:54.902+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:55.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:55.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:55.159+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:55.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:55.160+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:55.178+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:55.178+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:55.178+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:55.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:55.178+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:56.296 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:56.133+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:56.297 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:56.133+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:56.297 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:56.133+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:56.297 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:56.134+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:56.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:56.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:56.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:56.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:56.297 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:56.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:56.298 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:56.171+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:56.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:55.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:56.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:55.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:56.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:55.950+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:56.298 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:55.950+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:57.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:57.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:57.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:57.175+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:57.176+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:57.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:57.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:57.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:57.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:57.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:57.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:57.194+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:56.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:56.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:56.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:57.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:56.992+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:58.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:58.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:58.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:58.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:58.155+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:58.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:58.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:58.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:58.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:58.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:58.169+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:58.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:58.170+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:58.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:58.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:58.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:58.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:58.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:58.036+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:58.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:58.036+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:59.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:59.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:59.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:59.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:00:59.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:00:59.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:59.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:59.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:59.157+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:00:59.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:00:59.158+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:00:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:59.053+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:59.053+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:59.053+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:00:59.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:00:59.054+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:00:59.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:00:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:00:59.405+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21450 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:00.163+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:00.163+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:00.163+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:00.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:00.164+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:00.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:00.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:00.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:00.092+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:00.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:00.093+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:00.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:00.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:00.137+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:00.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:00.138+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:01.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:01.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:01.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:01.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:01.449 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:01.154+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:01.450 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:01.155+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:01.450 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:01.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:01.450 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:01.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:01.450 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:01.117+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:01.450 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:01.118+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:01.451 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:01.131+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:01.451 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:01.131+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:01.451 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:01.131+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:01.451 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:01.132+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:02.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:02.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:02.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:02.124+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:02.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:02.125+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:02.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:02.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:02.151+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:02.152+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:02.124+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:02.124+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:02.124+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:02.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:02.125+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:03.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:03.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:03.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:03.170+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:03.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:03.171+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:03.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:03.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:03.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:03.143+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:03.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:03.144+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:03.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:03.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:03.153+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:03.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:03.153+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:04.410 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:04.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:04.410 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:04.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:04.410 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:04.207+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:04.410 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:04.207+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:04.411 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:04.166+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:04.411 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:04.166+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:04.411 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:04.166+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:04.411 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:04.167+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:04.411 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:04.121+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:04.411 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:04.121+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:04.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:04.121+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:04.412 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:04.122+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:04.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:04.407+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21502 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:05.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:05.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:05.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:05.204+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:05.205+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:05.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:05.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:05.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:05.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:05.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:05.170+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:05.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:05.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:05.177+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:05.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:05.177+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:06.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:06.250+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:06.250+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:06.250+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:06.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:06.251+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:06.135+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:06.135+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:06.135+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:06.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:06.136+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:06.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:06.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:06.193+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:06.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:06.194+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:07.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:07.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:07.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:07.114+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:07.115+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:07.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:07.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:07.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:07.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:07.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:07.231+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:07.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:07.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:07.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:07.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:07.259+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:08.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:08.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:08.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:08.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:08.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:08.206+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:08.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:08.207+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:08.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:08.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:08.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:08.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:08.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:08.275+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:08.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:08.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:08.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:08.130+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:08.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:08.131+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:09.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:09.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:09.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:09.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:09.232 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:09.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:09.233 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:09.092+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:09.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:09.409+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21555 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:09.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:09.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:09.230+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:09.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:09.231+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:09.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:09.247+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:09.247+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:09.247+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:09.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:09.248+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:10.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:10.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:10.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:10.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:10.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:10.052+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:10.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:10.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:10.196+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:10.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:10.197+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:10.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:10.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:10.279+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:10.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:10.279+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:11.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:11.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:11.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:11.065+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:11.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:11.065+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:11.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:11.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:11.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:11.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:11.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:11.220+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:11.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:11.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:11.253+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:11.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:11.254+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:12.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:12.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:12.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:12.096+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:12.097+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:12.231+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:12.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:12.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:12.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:12.232+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:12.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:12.232+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:12.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:12.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:12.245+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:12.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:12.246+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:13.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:13.235+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:13.235+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:13.235+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:13.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:13.236+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:13.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:13.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:13.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:13.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:13.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:13.117+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:13.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:13.118+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:13.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:13.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:13.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:13.285+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:13.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:13.285+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:14.414 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:14.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:14.414 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:14.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:14.414 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:14.234+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:14.414 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:14.234+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:14.415 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:14.302+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:14.415 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:14.302+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:14.415 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:14.302+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:14.415 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:14.303+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:14.415 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:14.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:14.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:14.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:14.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:14.158+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:14.416 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:14.159+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:14.809 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:14.411+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21607 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:15.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:15.146+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:15.146+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:15.146+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:15.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:15.147+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:15.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:15.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:15.220+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:15.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:15.221+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:15.334+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:15.334+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:15.334+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:15.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:15.335+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:16.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:16.141+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:16.141+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:16.141+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:16.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:16.142+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:16.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:16.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:16.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:16.270+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:16.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:16.271+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:16.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:16.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:16.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:16.310+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:16.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:16.311+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:17.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:17.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:17.170+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:17.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:17.171+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:17.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:17.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:17.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:17.289+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:17.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:17.290+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:17.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:17.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:17.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:17.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:17.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:17.314+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:18.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:18.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:18.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:18.314+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:18.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:18.314+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:18.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:18.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:18.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:18.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:18.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:18.269+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:18.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:18.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:18.181+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:18.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:18.182+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:19.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:19.218+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:19.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:19.218+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:19.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:19.218+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:19.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:19.219+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:19.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:19.414+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21660 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:19.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:19.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:19.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:19.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:19.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:19.308+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:19.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:19.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:19.313+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:19.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:19.314+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:20.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:20.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:20.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:20.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:20.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:20.285+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:20.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:20.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:20.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:20.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:20.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:20.262+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:20.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:20.262+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:20.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:20.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:20.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:20.274+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:20.565 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:20.275+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:21.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:21.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:21.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:21.233+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:21.234+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:21.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:21.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:21.264+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:21.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:21.264+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:21.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:21.291+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:21.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:21.291+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:21.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:21.291+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:21.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:21.292+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:22.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:22.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:22.272+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:22.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:22.273+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:22.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:22.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:22.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:22.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:22.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:22.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:22.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:22.325+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:22.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:22.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:22.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:22.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:22.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:22.312+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:23.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:23.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:23.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:23.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:23.299+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:23.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:23.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:23.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:23.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:23.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:23.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:23.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:23.359+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:23.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:23.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:23.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:23.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:23.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:23.295+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:23.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:23.296+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:24.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:24.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:24.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:24.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:24.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:24.258+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:24.297 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:24.259+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:24.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:24.416+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21712 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:24.294+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:24.294+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:24.294+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:24.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:24.295+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:24.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:24.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:24.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:24.358+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:24.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:24.359+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:25.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:25.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:25.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:25.261+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:25.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:25.263+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:25.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:25.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:25.314+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:25.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:25.315+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:25.397+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:25.397+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:25.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:25.397+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:25.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:25.398+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:26.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:26.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:26.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:26.315+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:26.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:26.316+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:26.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:26.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:26.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:26.377+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:26.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:26.378+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:26.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:26.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:26.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:26.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:26.248+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:26.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:26.249+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:27.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:27.398+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:27.398+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:27.398+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:27.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:27.399+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:27.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:27.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:27.284+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:27.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:27.285+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:27.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:27.307+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:27.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:27.307+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:27.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:27.307+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:27.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:27.307+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:28.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:28.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:28.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:28.314+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:28.314+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:28.352+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:28.352+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:28.352+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:28.353+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:28.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:28.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:28.378+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:28.379+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:29.418+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21765 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:29.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:29.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:29.339+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:29.339+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:29.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:29.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:29.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:29.346+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:29.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:29.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:29.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:29.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:29.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:29.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:29.402+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:30.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:30.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:30.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:30.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:30.299+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:30.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:30.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:30.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:30.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:30.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:30.350+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:30.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:30.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:30.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:30.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:30.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:30.415+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:30.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:30.416+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:31.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:31.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:31.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:31.268+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:31.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:31.269+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:31.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:31.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:31.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:31.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:31.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:31.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:31.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:31.438+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:31.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:31.438+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:31.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:31.438+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:31.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:31.439+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:32.432+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:32.432+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:32.432+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:32.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:32.433+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:32.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:32.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:32.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:32.311+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:32.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:32.312+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:32.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:32.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:32.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:32.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:32.298+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:32.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:32.299+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:33.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:33.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:33.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:33.301+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:33.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:33.302+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:33.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:33.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:33.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:33.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:33.329+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:33.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:33.330+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:33.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:33.398+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:33.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:33.398+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:33.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:33.398+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:33.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:33.399+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:34.420+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21817 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:34.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:34.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:34.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:34.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:34.316+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:34.349+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:34.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:34.349+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:34.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:34.349+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:34.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:34.349+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:34.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:34.369+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:34.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:34.369+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:34.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:34.369+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:34.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:34.370+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:35.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:35.328+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:35.328+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:35.328+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:35.329+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:35.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:35.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:35.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:35.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:35.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:35.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:35.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:35.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:35.383+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:36.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:36.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:36.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:36.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:36.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:36.332+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:36.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:36.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:36.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:36.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:36.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:36.299+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:36.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:36.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:36.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:36.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:36.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:36.400+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:36.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:36.401+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:37.560 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:37.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:37.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:37.251+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:37.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:37.252+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:37.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:37.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:37.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:37.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:37.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:37.364+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:37.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:37.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:37.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:37.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:37.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:37.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:37.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:37.326+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:38.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:38.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:38.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:38.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:38.411+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:38.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:38.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:38.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:38.340+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:38.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:38.340+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:38.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:38.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:38.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:38.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:38.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:38.278+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:38.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:38.278+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:39.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:39.422+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21870 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:39.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:39.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:39.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:39.376+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:39.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:39.326+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:39.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:39.326+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:39.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:39.326+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:39.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:39.326+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:39.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:39.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:39.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:39.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:39.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:39.229+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:39.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:39.230+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:40.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:40.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:40.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:40.325+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:40.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:40.326+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:40.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:40.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:40.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:40.228+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:40.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:40.229+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:40.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:40.362+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:40.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:40.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:40.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:40.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:40.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:40.364+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:41.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:41.305+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:41.305+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:41.305+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:41.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:41.306+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:41.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:41.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:41.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:41.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:41.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:41.268+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:41.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:41.268+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:41.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:41.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:41.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:41.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:41.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:41.366+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:41.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:41.366+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:42.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:42.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:42.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:42.252+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:42.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:42.253+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:42.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:42.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:42.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:42.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:42.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:42.257+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:42.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:42.258+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:42.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:42.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:42.564 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:42.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:42.564 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:42.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:42.564 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:42.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:43.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:43.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:43.255+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:43.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:43.256+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:43.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:43.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:43.308+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:43.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:43.309+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:43.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:43.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:43.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:43.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:43.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:43.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:43.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:43.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:44.329 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:44.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:44.330 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:44.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:44.330 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:44.219+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:44.330 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:44.220+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:44.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:44.424+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21922 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:44.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:44.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:44.343+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:44.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:44.344+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:44.327+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:44.327+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:44.327+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:44.328+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:45.529 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:45.269+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:45.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:45.269+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:45.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:45.269+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:45.530 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:45.270+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:45.530 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:45.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:45.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:45.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:45.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:45.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:45.531 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:45.333+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:45.531 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:45.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:45.531 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:45.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:45.531 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:45.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:45.531 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:45.332+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:46.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:46.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:46.241+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:46.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:46.242+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:46.338+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:46.338+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:46.338+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:46.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:46.339+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:46.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:46.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:46.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:46.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:46.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:46.294+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:46.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:46.294+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:47.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:47.366+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:47.366+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:47.366+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:47.367+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:47.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:47.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:47.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:47.284+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:47.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:47.285+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:47.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:47.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:47.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:47.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:47.324+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:48.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:48.387+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:48.387+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:48.387+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:48.388+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:48.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:48.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:48.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:48.332+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:48.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:48.332+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:48.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:48.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:48.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:48.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:48.276+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:49.561 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:49.426+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 21975 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:49.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:49.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:49.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:49.374+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:49.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:49.375+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:49.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:49.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:49.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:49.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:49.323+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:49.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:49.323+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:49.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:49.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:49.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:49.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:49.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:49.274+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:49.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:49.274+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:50.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:50.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:50.363+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:50.363+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:50.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:50.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:50.315+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:50.316+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:50.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:50.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:50.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:50.341+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:50.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:50.342+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:51.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:51.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:51.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:51.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:51.390+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:51.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:51.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:51.271+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:51.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:51.272+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:51.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:51.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:51.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:51.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:51.387+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:51.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:51.388+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:52.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:52.388+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:52.388+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:52.388+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:52.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:52.388+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:52.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:52.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:52.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:52.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:52.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:52.302+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:52.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:52.303+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:52.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:52.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:52.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:52.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:52.396+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:52.563 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:52.397+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:53.385+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:53.385+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:53.385+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:53.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:53.386+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:53.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:53.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:53.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:53.439+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:53.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:53.440+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:53.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:53.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:53.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:53.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:53.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:53.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:53.563 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:53.298+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:54.745 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:54.428+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22027 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:54.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:54.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:54.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:54.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:54.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:54.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:54.746 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:54.337+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:54.747 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:54.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:54.747 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:54.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:54.747 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:54.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:54.747 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:54.398+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:54.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:54.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:54.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:54.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:54.748 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:54.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:54.748 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:54.446+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:55.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:55.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:55.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:55.377+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:55.378+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:55.321+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:55.321+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:55.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:55.321+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:55.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:55.322+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:55.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:55.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:55.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:55.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:55.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:55.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:55.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:55.452+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:56.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:56.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:56.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:56.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:56.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:56.427+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:56.427+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:56.427+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:56.428+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:56.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:56.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:56.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:56.334+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:57.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:57.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:57.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:57.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:57.450+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:57.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:57.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:57.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:57.389+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:57.390+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:57.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:57.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:57.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:57.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:57.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:57.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:57.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:57.345+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:58.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:58.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:58.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:58.383+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:58.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:58.384+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:58.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:58.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:58.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:58.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:58.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:58.435+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:58.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:58.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:58.298+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:58.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:58.299+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:59.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:01:59.430+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22080 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:01:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:59.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:59.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:59.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:01:59.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:01:59.446+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:01:59.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:59.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:59.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:59.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:59.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:59.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:01:59.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:01:59.295+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:01:59.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:59.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:59.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:59.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:59.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:59.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:01:59.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:01:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:01:59.413+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:00.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:00.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:00.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:00.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:00.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:00.463+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:00.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:00.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:00.310+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:00.311+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:00.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:00.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:00.460+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:00.461+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:01.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:01.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:01.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:01.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:01.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:01.450+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:01.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:01.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:01.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:01.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:01.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:01.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:01.480+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:01.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:01.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:01.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:01.351+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:02.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:02.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:02.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:02.474+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:02.475+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:02.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:02.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:02.460+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:02.461+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:02.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:02.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:02.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:02.346+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:03.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:03.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:03.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:03.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:03.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:03.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:03.659 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:03.485+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:03.659 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:03.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:03.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:03.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:03.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:03.379+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:03.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:03.379+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:03.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:03.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:03.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:03.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:03.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:03.511+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:03.661 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:03.512+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:04.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:04.432+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22132 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:04.523+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:04.523+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:04.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:04.523+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:04.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:04.523+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:04.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:04.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:04.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:04.478+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:04.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:04.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:04.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:04.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:04.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:04.336+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:04.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:04.337+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:05.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:05.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:05.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:05.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:05.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:05.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:05.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:05.533+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:05.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:05.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:05.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:05.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:05.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:05.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:05.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:05.520+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:05.816 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:05.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:05.816 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:05.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:05.816 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:05.325+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:05.816 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:05.326+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:06.538 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:06.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:06.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:06.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:06.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:06.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:06.539 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:06.499+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:06.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:06.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:06.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:06.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:06.539 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:06.285+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:06.540 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:06.286+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:06.540 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:06.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:06.540 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:06.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:06.540 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:06.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:06.540 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:06.536+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:07.546 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:07.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:07.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:07.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:07.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:07.294+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:07.547 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:07.295+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:07.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:07.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:07.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:07.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:07.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:07.545+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:07.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:07.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:07.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:07.547+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:08.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:08.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:08.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:08.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:08.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:08.322+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:08.544 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:08.323+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:08.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:08.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:08.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:08.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:08.549+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:08.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:08.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:08.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:08.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:08.543+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:09.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:09.434+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22185 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:09.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:09.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:09.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:09.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:09.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:09.535+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:09.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:09.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:09.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:09.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:09.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:09.352+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:09.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:09.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:09.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:09.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:09.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:09.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:09.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:09.510+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:10.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:10.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:10.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:10.583+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:10.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:10.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:10.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:10.465+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:10.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:10.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:10.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:10.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:10.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:10.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:10.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:10.355+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:11.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:11.535+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:11.535+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:11.535+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:11.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:11.536+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:11.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:11.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:11.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:11.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:11.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:11.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:11.487+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:11.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:11.387+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:11.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:11.387+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:11.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:11.387+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:11.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:11.388+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:12.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:12.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:12.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:12.408+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:12.409+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:12.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:12.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:12.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:12.515+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:12.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:12.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:12.449+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:12.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:12.450+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:13.659 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:13.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:13.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:13.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:13.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:13.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:13.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:13.403+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:13.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:13.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:13.660 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:13.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:13.661 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:13.485+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:13.661 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:13.486+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:13.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:13.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:13.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:13.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:13.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:13.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:13.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:13.538+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:14.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:14.436+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22237 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:14.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:14.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:14.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:14.543+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:14.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:14.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:14.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:14.403+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:14.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:14.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:14.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:14.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:14.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:14.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:14.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:14.524+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:15.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:15.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:15.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:15.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:15.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:15.431+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:15.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:15.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:15.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:15.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:15.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:15.543+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:15.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:15.544+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:15.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:15.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:15.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:15.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:15.564+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:16.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:16.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:16.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:16.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:16.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:16.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:16.512 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:16.392+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:16.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:16.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:16.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:16.509+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:16.510+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:16.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:16.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:16.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:16.518+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:17.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:17.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:17.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:17.359+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:17.360+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:17.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:17.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:17.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:17.476+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:17.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:17.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:17.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:17.542+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:18.488 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:18.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:18.488 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:18.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:18.489 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:18.388+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:18.489 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:18.389+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:18.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:18.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:18.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:18.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:18.559+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:18.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:18.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:18.486+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:18.486+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:19.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:19.438+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22289 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:19.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:19.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:19.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:19.608+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:19.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:19.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:19.399+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:19.400+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:19.506+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:19.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:19.506+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:19.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:19.506+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:19.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:19.507+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:20.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:20.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:20.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:20.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:20.645 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:20.522+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:20.646 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:20.522+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:20.646 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:20.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:20.646 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:20.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:20.646 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:20.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:20.646 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:20.358+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:21.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:20.642+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:20.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:20.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:20.643+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:21.605 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:21.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:21.605 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:21.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:21.605 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:21.347+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:21.605 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:21.348+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:21.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:21.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:21.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:21.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:21.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:21.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:21.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:21.517+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:21.606 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:21.602+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:21.606 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:21.602+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:21.607 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:21.602+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:22.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:21.603+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:22.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:22.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:22.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:22.344+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:22.345+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:22.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:22.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:22.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:22.585+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:22.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:22.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:22.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:22.471+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:23.496 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:23.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:23.496 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:23.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:23.496 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:23.390+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:23.496 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:23.390+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:23.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:23.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:23.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:23.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:23.493+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:23.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:23.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:23.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:23.629+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:24.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:24.439+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22340 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:24.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:24.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:24.349+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:24.350+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:24.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:24.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:24.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:24.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:24.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:24.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:24.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:24.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:24.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:24.600+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:25.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:25.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:25.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:25.477+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:25.478+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:25.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:25.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:25.396+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:25.397+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:25.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:25.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:25.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:25.601+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:26.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:26.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:26.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:26.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:26.418+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:26.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:26.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:26.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:26.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:26.613+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:26.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:26.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:26.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:26.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:26.513+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:27.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:27.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:27.407+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:27.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:27.408+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:27.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:27.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:27.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:27.570+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:27.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:27.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:27.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:27.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:28.528 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:28.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:28.528 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:28.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:28.528 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:28.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:28.528 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:28.455+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:28.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:28.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:28.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:28.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:28.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:28.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:28.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:28.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:28.526+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:29.442+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22391 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:29.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:29.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:29.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:29.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:29.448+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:29.554+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:29.554+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:29.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:29.554+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:29.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:29.555+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:29.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:29.496+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:29.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:29.496+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:29.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:29.496+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:29.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:29.497+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:30.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:30.460+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:30.460+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:30.460+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:30.461+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:30.568+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:30.568+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:30.568+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:30.569+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:30.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:30.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:30.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:30.470+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:31.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:31.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:31.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:31.446+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:31.447+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:31.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:31.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:31.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:31.431+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:31.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:31.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:31.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:31.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:31.535+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:32.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:32.514+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:32.514+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:32.514+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:32.515+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:32.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:32.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:32.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:32.489+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:32.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:32.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:32.454+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:32.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:32.455+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:33.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:33.482+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:33.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:33.482+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:33.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:33.482+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:33.661 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:33.483+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:33.662 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:33.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:33.662 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:33.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:33.662 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:33.428+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:33.662 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:33.429+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:33.662 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:33.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:33.663 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:33.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:33.663 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:33.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:33.663 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:33.545+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:34.445+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22442 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:34.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:34.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:34.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:34.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:34.570+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:34.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:34.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:34.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:34.477+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:34.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:34.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:34.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:34.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:34.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:34.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:34.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:34.484+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:35.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:35.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:35.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:35.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:35.481+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:35.488+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:35.488+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:35.488+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:35.489+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:35.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:35.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:35.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:35.557+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:36.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:36.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:36.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:36.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:36.502+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:36.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:36.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:36.569+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:36.569+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:36.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:36.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:36.457+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:36.458+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:37.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:37.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:37.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:37.498+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:37.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:37.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:37.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:37.509+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:37.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:37.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:37.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:37.525+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:37.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:37.525+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:38.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:38.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:38.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:38.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:38.566+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:38.486+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:38.486+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:38.486+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:38.487+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:38.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:38.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:38.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:38.533+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:39.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:39.447+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22495 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:39.496+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:39.496+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:39.496+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:39.497+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:39.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:39.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:39.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:39.525+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:39.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:39.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:39.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:39.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:39.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:39.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:39.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:39.495+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:40.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:40.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:40.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:40.525+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:40.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:40.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:40.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:40.472+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:40.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:40.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:40.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:40.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:40.538+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:41.512+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:41.512+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:41.512+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:41.513+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:41.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:41.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:41.483+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:41.484+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:41.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:41.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:41.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:41.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:41.557+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:42.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:42.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:42.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:42.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:42.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:42.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:42.815 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:42.597+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:42.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:42.540+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:42.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:42.540+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:42.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:42.540+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:42.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:42.541+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:42.816 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:42.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:42.816 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:42.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:42.816 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:42.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:42.816 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:42.507+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:43.662 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:43.517+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:43.662 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:43.517+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:43.662 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:43.517+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:43.663 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:43.518+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:43.663 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:43.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:43.663 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:43.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:43.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:43.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:43.664 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:43.577+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:43.664 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:43.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:43.665 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:43.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:43.665 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:43.500+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:43.665 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:43.501+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:44.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:44.449+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22547 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:44.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:44.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:44.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:44.501+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:44.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:44.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:44.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:44.581+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:44.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:44.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:44.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:44.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:44.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:44.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:44.552+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:45.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:45.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:45.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:45.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:45.555+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:45.479+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:45.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:45.479+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:45.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:45.479+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:45.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:45.480+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:45.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:45.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:45.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:45.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:45.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:45.570+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:46.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:46.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:46.602+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:46.602+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:46.602+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:46.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:46.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:46.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:46.594+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:46.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:46.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:46.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:46.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:46.490+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:47.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:47.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:47.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:47.521+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:47.522+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:47.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:47.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:47.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:47.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:47.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:47.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:47.585+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:47.545+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:47.545+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:47.545+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:47.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:47.546+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:48.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:48.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:48.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:48.571+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:48.572+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:48.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:48.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:48.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:48.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:48.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:48.607+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:48.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:48.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:48.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:48.515+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:49.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:49.451+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22600 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:49.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:49.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:49.628+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:49.629+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:49.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:49.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:49.513+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:49.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:49.514+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:49.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:49.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:49.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:49.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:49.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:49.613+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:50.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:50.509+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:50.509+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:50.509+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:50.510+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:50.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:50.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:50.646+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:50.647+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:50.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:50.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:50.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:50.661+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:51.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:51.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:51.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:51.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:51.643+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:51.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:51.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:51.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:51.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:51.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:51.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:51.477+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:51.478+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:52.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:52.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:52.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:52.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:52.655+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:52.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:52.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:52.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:52.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:52.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:52.526+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:52.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:52.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:52.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:52.581+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:53.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:53.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:53.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:53.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:53.576+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:53.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:53.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:53.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:53.532+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:53.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:53.532+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:53.704+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:53.704+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:53.704+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:53.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:53.704+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:54.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:54.453+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22652 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:54.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:54.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:54.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:54.545+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:54.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:54.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:54.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:54.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:54.687+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:54.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:54.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:54.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:54.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:54.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:54.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:54.589+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:56.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:55.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:55.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:55.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:55.603+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:55.591+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:55.591+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:55.591+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:55.592+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:55.710+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:55.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:55.711+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:55.711+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:56.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:56.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:56.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:56.575+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:56.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:56.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:56.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:56.713+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:56.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:56.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:57.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:56.571+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:57.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:56.572+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:58.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:57.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:57.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:57.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:57.717+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:57.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:57.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:57.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:57.601+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:57.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:57.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:57.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:58.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:57.569+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:58.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:58.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:58.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:58.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:58.600+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:58.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:58.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:58.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:02:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:58.724+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:02:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:58.636+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:58.636+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:58.636+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:58.636+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:59.765 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:02:59.455+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22703 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:02:59.766 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:59.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:59.766 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:59.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:59.767 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:59.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:02:59.767 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:02:59.617+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:02:59.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:59.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:59.767 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:59.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:59.768 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:59.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:02:59.768 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:02:59.626+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:00.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:59.763+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:59.763+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:59.763+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:00.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:02:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:02:59.763+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:01.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:00.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:00.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:00.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:00.601+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:00.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:00.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:00.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:00.813+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:00.645+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:00.645+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:00.645+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:00.646+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:02.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:01.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:01.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:01.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:01.772+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:01.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:01.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:01.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:01.685+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:01.565+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:01.565+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:01.565+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:01.565+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:02.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:02.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:02.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:02.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:02.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:02.525+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:02.745+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:02.745+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:02.745+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:02.746+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:02.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:02.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:02.729+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:02.730+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:03.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:03.721+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:03.721+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:03.721+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:03.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:03.722+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:03.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:03.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:03.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:03.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:03.552+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:03.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:03.553+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:03.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:03.735+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:03.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:03.735+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:03.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:03.735+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:03.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:03.736+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:04.721 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:04.457+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22756 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:04.721 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:04.517+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:04.721 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:04.517+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:04.721 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:04.517+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:04.722 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:04.518+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:05.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:04.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:04.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:04.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:04.719+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:04.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:04.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:04.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:05.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:04.745+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:05.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:05.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:05.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:05.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:05.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:05.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:05.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:05.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:06.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:05.707+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:05.707+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:05.707+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:05.708+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:05.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:05.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:05.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:05.784+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:06.584+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:06.584+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:06.584+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:07.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:06.585+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:06.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:07.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:06.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:07.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:06.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:07.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:06.662+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:06.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:06.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:07.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:06.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:07.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:06.767+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:07.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:08.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:07.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:08.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:07.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:08.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:07.738+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:08.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:07.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:08.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:07.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:08.065 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:07.634+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:08.065 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:07.635+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:08.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:07.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:08.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:07.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:08.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:07.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:08.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:07.707+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:08.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:08.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:08.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:08.720+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:08.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:08.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:08.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:08.977 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:08.744+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:08.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:08.619+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:08.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:08.619+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:08.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:08.619+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:08.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:08.620+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:09.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:09.459+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22808 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:09.696+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:09.696+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:09.696+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:09.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:09.697+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:09.676+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:09.676+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:09.676+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:09.677+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:09.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:09.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:09.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:09.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:09.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:09.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:09.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:09.589+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:11.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:10.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:11.090 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:10.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:11.090 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:10.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:11.090 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:10.680+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:11.091 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:10.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:11.091 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:10.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:11.091 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:10.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:11.091 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:10.600+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:11.092 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:10.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:11.092 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:10.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:11.092 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:10.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:11.092 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:10.746+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:12.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:11.597+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:11.597+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:11.597+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:11.598+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:11.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:11.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:11.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:11.771+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:11.633+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:11.633+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:11.633+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:11.634+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:13.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:12.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:12.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:12.627+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:12.627+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:12.580+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:12.580+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:12.580+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:12.581+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:12.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:12.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:12.740+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:12.741+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:14.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:13.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:13.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:13.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:13.784+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:13.585+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:13.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:13.586+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:13.587+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:13.580+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:13.580+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:13.580+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:13.581+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:14.772 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:14.461+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22861 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:14.773 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:14.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:14.773 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:14.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:14.773 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:14.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:14.773 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:14.616+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:14.773 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:14.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:14.774 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:14.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:14.774 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:14.551+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:14.774 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:14.551+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:15.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:14.770+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:14.770+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:14.770+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:14.771+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:16.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:15.581+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:15.581+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:15.581+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:15.582+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:15.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:15.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:16.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:15.571+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:16.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:15.571+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:15.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:15.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:15.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:16.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:15.756+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:16.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:16.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:16.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:16.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:16.718+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:16.543+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:16.543+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:16.543+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:16.544+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:16.611+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:16.611+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:16.611+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:16.612+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:17.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:17.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:17.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:17.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:17.537+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:17.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:17.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:17.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:17.733+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:17.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:17.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:17.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:17.642+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:18.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:18.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:18.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:18.713+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:18.663+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:18.663+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:18.663+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:18.663+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:18.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:18.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:18.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:18.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:18.502+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:19.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:19.464+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22913 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:19.614+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:19.614+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:19.614+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:19.614+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:19.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:19.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:19.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:19.673+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:19.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:19.455+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:19.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:19.455+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:19.455+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:19.456+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:20.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:20.663+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:20.663+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:20.663+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:20.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:20.664+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:20.658+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:20.658+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:20.658+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:20.659+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:20.427+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:20.427+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:20.427+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:20.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:20.428+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:21.701 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:21.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:21.701 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:21.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:21.702 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:21.401+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:21.702 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:21.402+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:21.702 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:21.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:21.702 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:21.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:21.702 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:21.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:21.702 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:21.647+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:22.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:21.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:21.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:21.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:21.699+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:22.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:22.352+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:22.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:22.352+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:22.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:22.353+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:22.658 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:22.354+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:22.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:22.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:22.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:22.656+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:22.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:22.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:22.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:23.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:22.736+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:23.665 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:23.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:23.666 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:23.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:23.666 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:23.379+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:23.666 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:23.380+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:23.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:23.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:23.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:23.669+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:23.739+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:23.739+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:23.739+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:23.740+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:24.698 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:24.466+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 22966 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:24.698 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:24.354+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:24.698 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:24.354+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:24.698 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:24.354+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:24.699 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:24.354+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:25.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:24.695+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:24.695+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:24.695+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:24.696+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:24.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:24.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:24.777+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:24.778+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:25.709 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:25.375+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:25.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:25.375+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:25.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:25.375+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:25.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:25.376+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:26.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:25.707+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:25.707+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:25.707+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:25.708+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:25.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:25.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:25.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:25.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:26.757 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:26.425+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:26.758 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:26.425+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:26.758 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:26.425+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:26.758 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:26.426+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:27.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:26.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:26.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:26.755+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:26.756+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:26.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:26.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:26.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:26.768+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:27.762 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:27.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:27.762 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:27.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:27.762 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:27.437+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:27.762 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:27.438+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:27.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:27.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:27.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:27.792+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:27.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:27.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:27.760+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:27.760+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:28.724 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:28.419+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:28.724 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:28.419+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:28.724 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:28.419+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:28.725 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:28.420+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:28.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:28.722+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:28.722+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:28.722+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:28.722+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:28.787+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:28.787+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:28.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:28.787+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:28.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:28.788+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:29.468+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23018 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:29.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:29.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:29.675+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:29.676+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:29.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:29.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:29.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:29.463+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:30.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:29.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:29.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:29.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:29.825+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:30.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:30.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:30.493+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:30.494+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:30.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:30.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:30.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:30.653+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:31.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:30.842+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:30.843+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:30.843+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:30.843+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:31.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:31.505+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:31.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:31.507+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:31.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:31.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:31.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:31.618+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:31.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:31.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:31.889+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:31.890+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:32.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:32.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:32.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:32.554+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:32.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:32.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:32.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:32.612+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:32.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:32.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:32.913+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:32.914+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:33.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:33.509+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:33.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:33.509+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:33.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:33.509+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:33.787 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:33.509+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:33.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:33.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:33.788 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:33.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:33.788 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:33.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:33.788 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:33.630+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:34.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:33.941+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:34.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:33.941+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:34.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:33.941+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:34.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:33.941+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:34.470+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23071 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:34.496+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:34.496+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:34.496+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:34.496+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:34.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:34.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:34.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:34.677+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:35.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:34.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:34.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:34.905+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:34.906+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:35.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:35.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:35.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:35.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:35.638+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:35.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:35.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:35.514+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:35.515+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:36.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:35.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:36.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:35.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:36.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:35.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:36.121 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:35.859+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:36.830 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:36.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:36.830 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:36.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:36.831 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:36.648+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:36.831 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:36.649+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:36.831 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:36.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:36.831 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:36.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:36.831 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:36.564+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:36.832 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:36.564+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:37.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:36.828+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:37.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:36.828+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:37.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:36.828+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:37.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:36.829+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:37.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:37.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:37.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:37.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:37.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:37.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:37.789 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:37.603+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:37.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:37.522+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:37.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:37.522+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:37.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:37.522+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:37.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:37.523+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:38.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:37.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:37.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:37.786+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:37.787+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:38.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:38.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:38.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:38.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:38.640+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:38.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:38.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:38.558+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:38.558+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:38.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:38.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:38.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:38.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:38.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:38.767+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:39.805 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:39.472+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23123 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:39.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:39.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:39.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:39.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:39.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:39.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:39.805 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:39.673+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:39.806 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:39.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:39.806 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:39.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:39.806 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:39.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:39.806 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:39.582+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:40.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:39.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:39.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:39.802+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:40.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:39.803+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:41.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:40.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:40.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:40.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:41.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:40.647+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:40.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:41.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:40.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:40.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:40.799+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:41.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:40.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:41.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:40.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:41.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:40.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:41.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:40.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:42.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:41.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:41.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:41.645+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:41.646+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:41.755+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:42.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:41.755+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:41.755+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:41.756+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:42.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:41.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:42.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:41.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:42.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:41.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:42.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:41.671+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:43.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:42.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:42.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:42.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:42.683+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:42.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:42.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:42.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:42.685+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:42.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:42.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:42.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:42.784+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:43.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:43.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:43.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:43.747+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:43.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:43.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:43.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:43.640+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:43.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:43.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:44.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:43.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:44.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:43.661+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:44.769 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:44.474+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23176 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:44.769 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:44.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:44.769 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:44.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:44.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:44.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:44.770 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:44.666+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:44.770 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:44.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:44.770 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:44.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:44.770 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:44.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:44.770 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:44.629+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:45.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:44.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:44.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:44.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:44.768+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:46.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:45.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:45.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:45.765+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:45.766+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:45.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:46.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:45.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:46.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:45.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:46.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:45.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:45.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:45.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:45.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:45.677+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:47.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:46.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:46.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:46.745+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:46.745+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:46.698+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:46.698+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:46.698+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:46.699+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:47.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:46.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:47.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:46.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:47.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:46.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:47.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:46.623+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:47.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:47.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:47.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:47.735+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:47.742+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:47.742+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:47.742+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:47.743+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:48.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:47.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:48.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:47.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:48.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:47.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:48.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:47.664+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:48.774+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:48.774+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:48.774+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:48.775+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:48.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:48.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:48.758+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:48.758+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:48.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:48.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:48.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:48.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:48.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:48.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:48.686+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:49.765 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:49.476+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23228 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:49.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:49.688+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:49.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:49.688+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:49.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:49.688+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:49.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:49.689+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:49.775+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:49.775+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:49.775+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:49.776+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:49.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:49.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:49.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:49.764+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:51.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:50.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:50.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:50.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:50.646+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:50.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:50.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:51.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:50.783+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:51.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:50.784+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:50.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:50.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:50.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:50.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:52.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:51.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:51.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:51.695+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:51.695+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:51.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:51.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:51.752+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:51.753+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:51.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:51.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:52.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:51.784+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:52.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:51.785+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:53.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:52.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:52.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:52.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:52.745+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:52.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:53.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:52.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:53.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:52.761+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:53.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:52.761+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:52.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:52.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:52.814+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:53.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:52.815+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:53.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:53.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:53.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:53.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:53.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:53.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:53.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:53.772+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:53.944 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:53.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:53.944 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:53.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:53.945 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:53.788+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:53.945 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:53.788+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:53.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:53.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:53.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:53.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:53.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:53.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:53.945 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:53.746+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:54.792 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:54.479+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23281 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:03:54.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:54.698+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:54.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:54.698+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:54.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:54.698+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:54.792 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:54.699+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:55.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:54.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:55.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:54.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:54.789+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:54.790+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:54.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:54.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:54.801+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:54.802+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:56.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:55.804+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:55.804+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:55.804+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:55.805+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:55.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:55.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:55.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:55.740+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:55.766+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:55.766+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:55.766+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:56.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:55.766+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:56.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:56.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:56.771+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:56.771+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:56.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:56.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:56.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:56.811+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:56.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:56.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:57.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:56.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:57.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:56.785+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:58.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:57.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:57.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:57.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:57.750+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:57.824+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:57.824+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:57.824+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:57.824+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:57.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:57.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:57.752+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:57.752+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:58.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:58.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:58.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:58.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:58.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:58.858+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:03:58.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:58.859+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:03:58.983 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:58.742+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:58.983 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:58.742+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:58.983 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:58.742+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:03:58.984 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:58.742+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:58.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:58.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:58.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:58.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:58.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:58.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:03:58.984 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:58.724+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:03:59.745 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:03:59.480+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23333 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:00.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:59.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:59.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:59.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:03:59.872+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:59.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:59.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:59.753+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:00.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:03:59.754+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:59.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:59.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:59.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:00.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:03:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:03:59.744+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:01.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:00.896+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:00.896+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:00.896+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:00.897+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:00.709+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:00.709+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:00.709+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:00.709+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:00.738+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:00.738+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:01.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:00.738+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:01.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:00.739+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:01.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:01.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:01.717+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:01.717+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:01.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:01.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:01.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:01.671+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:01.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:01.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:01.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:02.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:01.869+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:03.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:02.823+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:02.823+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:02.823+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:02.824+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:02.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:03.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:02.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:03.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:02.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:03.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:02.687+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:02.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:02.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:03.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:02.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:03.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:02.743+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:04.031 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:03.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:04.031 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:03.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:04.031 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:03.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:04.031 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:03.713+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:04.032 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:03.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:04.032 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:03.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:04.032 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:03.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:04.032 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:03.659+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:04.032 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:03.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:04.032 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:03.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:04.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:03.856+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:04.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:03.856+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:04.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:04.483+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23386 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:04.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:04.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:04.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:04.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:04.665+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:04.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:04.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:04.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:04.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:04.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:04.640+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:04.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:04.641+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:05.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:04.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:04.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:04.862+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:04.863+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:06.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:05.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:05.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:05.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:05.842+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:05.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:05.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:05.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:05.607+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:05.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:05.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:05.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:05.655+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:06.870 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:06.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:06.870 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:06.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:06.871 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:06.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:06.871 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:06.671+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:06.871 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:06.601+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:06.871 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:06.601+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:06.871 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:06.601+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:06.871 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:06.601+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:07.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:06.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:06.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:06.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:06.869+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:07.880 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:07.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:07.880 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:07.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:07.880 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:07.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:07.880 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:07.621+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:07.881 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:07.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:07.881 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:07.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:07.881 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:07.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:07.881 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:07.673+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:08.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:07.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:07.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:07.877+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:08.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:07.878+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:08.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:08.646+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:08.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:08.646+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:08.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:08.646+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:08.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:08.647+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:08.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:08.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:08.846+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:08.846+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:08.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:08.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:08.612+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:08.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:08.613+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:09.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:09.486+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23438 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:09.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:09.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:09.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:09.631+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:09.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:09.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:09.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:09.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:09.645+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:09.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:09.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:09.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:09.810+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:11.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:10.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:10.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:10.651+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:10.651+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:10.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:10.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:10.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:10.654+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:10.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:10.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:10.845+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:10.846+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:12.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:11.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:11.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:11.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:11.661+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:11.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:11.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:11.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:12.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:11.664+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:11.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:11.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:11.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:12.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:11.840+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:13.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:12.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:12.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:12.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:12.671+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:12.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:12.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:12.841+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:12.842+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:12.659+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:12.659+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:12.659+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:12.660+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:14.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:13.842+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:13.842+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:13.842+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:13.843+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:13.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:13.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:13.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:14.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:13.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:13.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:13.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:13.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:13.653+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:14.802 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:14.488+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23491 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:14.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:14.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:14.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:14.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:14.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:14.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:14.803 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:14.685+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:14.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:14.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:14.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:14.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:14.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:14.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:14.803 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:14.663+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:15.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:14.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:14.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:14.800+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:14.800+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:16.057 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:15.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:16.057 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:15.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:16.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:15.794+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:16.058 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:15.795+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:16.058 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:15.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:16.058 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:15.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:16.058 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:15.649+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:16.059 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:15.651+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:16.059 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:15.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:16.059 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:15.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:16.059 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:15.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:16.059 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:15.648+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:16.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:16.622+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:16.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:16.623+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:16.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:17.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:16.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:17.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:16.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:17.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:16.631+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:16.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:16.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:16.766+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:17.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:16.767+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:18.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:17.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:17.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:17.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:17.597+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:17.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:17.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:17.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:17.601+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:17.754+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:17.754+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:17.754+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:18.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:17.755+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:18.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:18.573+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:18.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:18.573+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:18.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:18.573+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:18.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:18.574+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:18.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:18.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:18.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:18.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:18.977 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:18.647+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:18.977 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:18.647+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:18.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:18.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:18.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:18.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:18.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:18.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:18.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:18.719+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:19.749 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:19.490+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23543 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:19.749 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:19.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:19.749 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:19.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:19.749 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:19.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:19.750 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:19.600+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:19.750 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:19.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:19.750 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:19.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:19.750 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:19.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:19.750 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:19.525+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:20.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:19.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:19.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:19.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:19.747+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:20.736 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:20.476+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:20.736 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:20.476+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:20.736 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:20.476+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:20.736 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:20.477+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:20.737 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:20.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:20.737 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:20.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:20.737 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:20.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:20.737 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:20.618+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:21.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:20.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:20.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:20.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:21.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:20.734+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:21.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:21.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:21.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:21.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:21.505+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:21.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:21.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:21.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:21.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:21.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:21.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:21.655+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:21.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:21.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:21.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:21.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:21.707+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:22.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:22.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:22.680+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:22.681+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:22.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:22.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:22.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:22.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:22.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:22.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:22.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:22.481+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:22.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:22.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:22.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:22.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:22.814 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:22.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:22.814 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:22.624+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:23.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:23.479+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:23.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:23.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:23.480+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:23.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:23.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:23.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:23.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:23.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:23.671+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:23.638+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:23.638+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:23.638+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:23.639+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:24.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:24.493+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23596 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:24.702+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:24.702+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:24.702+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:24.702+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:24.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:24.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:24.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:24.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:24.460+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:24.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:24.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:24.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:24.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:24.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:24.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:24.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:24.600+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:25.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:25.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:25.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:25.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:25.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:25.497+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:26.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:25.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:25.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:25.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:25.743+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:25.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:25.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:25.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:25.565+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:26.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:26.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:26.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:26.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:26.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:26.525+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:26.719+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:26.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:26.719+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:26.719+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:26.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:26.720+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:26.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:26.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:26.527+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:26.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:26.528+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:27.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:27.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:27.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:27.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:27.507+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:27.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:27.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:27.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:27.733+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:27.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:27.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:27.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:27.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:27.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:27.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:27.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:27.517+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:28.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:28.463+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:28.463+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:28.463+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:28.464+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:28.703+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:28.703+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:28.703+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:28.705+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:28.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:28.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:28.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:28.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:29.497 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:29.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:29.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:29.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:29.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:29.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:29.498 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:29.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:29.748 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:29.495+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23648 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:29.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:29.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:29.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:29.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:29.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:29.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:29.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:29.509+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:30.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:29.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:29.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:29.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:29.747+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:30.712 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:30.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:30.712 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:30.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:30.713 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:30.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:30.713 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:30.526+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:30.713 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:30.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:30.713 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:30.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:30.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:30.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:30.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:30.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:31.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:30.709+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:30.710+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:30.710+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:30.711+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:31.493 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:31.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:31.494 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:31.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:31.494 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:31.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:31.494 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:31.491+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:31.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:31.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:31.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:31.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:31.687+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:31.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:31.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:31.491+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:31.492+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:32.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:32.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:32.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:32.643+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:32.499+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:32.499+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:32.499+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:32.499+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:32.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:32.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:32.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:32.469+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:33.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:33.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:33.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:33.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:33.636+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:33.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:33.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:33.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:33.477+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:33.499+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:33.499+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:33.499+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:33.499+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:34.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:34.497+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23701 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:34.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:34.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:34.502+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:34.503+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:34.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:34.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:34.433+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:34.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:34.434+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:35.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:34.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:34.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:34.608+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:34.609+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:35.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:35.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:35.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:35.650+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:35.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:35.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:35.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:35.426+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:35.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:35.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:35.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:35.521+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:36.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:36.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:36.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:36.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:36.690+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:36.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:36.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:36.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:36.461+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:36.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:36.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:36.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:36.570+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:37.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:37.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:37.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:37.719+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:37.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:37.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:37.500+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:37.501+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:37.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:37.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:37.525+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:37.526+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:38.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:38.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:38.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:38.521+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:38.521+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:38.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:38.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:38.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:38.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:38.538+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:38.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:38.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:38.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:38.727+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:39.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:39.500+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23753 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:39.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:39.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:39.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:39.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:39.536+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:40.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:39.704+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:39.704+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:39.704+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:39.705+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:39.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:40.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:39.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:40.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:39.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:40.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:39.583+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:40.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:40.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:40.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:40.618+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:40.619+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:40.703+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:40.703+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:40.703+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:40.703+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:40.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:40.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:40.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:40.555+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:42.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:41.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:41.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:41.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:41.634+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:41.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:41.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:41.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:41.689+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:41.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:41.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:41.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:41.575+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:43.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:42.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:42.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:42.599+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:43.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:42.599+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:42.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:42.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:42.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:43.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:42.644+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:42.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:42.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:42.657+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:42.658+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:44.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:43.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:43.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:43.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:43.576+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:43.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:43.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:43.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:44.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:43.607+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:43.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:43.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:43.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:43.670+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:44.610 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:44.502+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23806 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:44.611 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:44.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:44.611 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:44.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:44.611 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:44.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:44.611 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:44.586+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:45.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:44.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:44.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:44.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:44.609+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:44.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:44.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:45.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:44.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:45.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:44.663+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:45.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:45.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:45.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:45.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:45.550+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:45.643+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:45.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:45.643+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:45.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:45.643+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:45.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:45.644+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:45.620+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:45.620+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:45.620+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:45.620+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:47.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:46.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:46.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:46.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:46.580+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:46.666+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:46.666+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:46.666+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:46.667+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:46.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:46.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:47.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:46.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:47.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:46.577+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:47.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:47.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:47.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:47.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:47.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:47.560+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:47.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:47.561+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:47.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:47.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:47.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:47.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:47.557+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:47.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:47.558+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:47.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:47.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:47.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:47.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:47.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:47.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:47.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:47.634+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:48.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:48.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:48.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:48.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:48.615+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:48.616+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:48.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:48.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:48.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:48.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:48.564+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:48.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:48.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:48.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:48.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:48.591+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:49.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:49.504+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23858 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:49.565+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:49.565+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:49.565+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:49.566+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:49.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:49.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:49.544+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:49.545+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:49.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:49.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:49.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:49.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:49.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:49.576+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:51.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:50.601+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:50.601+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:50.601+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:51.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:50.601+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:50.626+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:50.626+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:51.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:50.626+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:51.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:50.627+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:50.563+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:50.563+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:50.563+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:50.564+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:51.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:51.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:51.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:51.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:51.537+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:51.594+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:51.594+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:51.594+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:51.595+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:51.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:51.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:51.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:51.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:51.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:51.638+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:51.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:51.638+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:52.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:52.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:52.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:52.663+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:52.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:52.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:52.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:52.552+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:52.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:52.574+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:52.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:52.574+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:52.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:52.574+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:52.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:52.575+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:53.683+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:53.683+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:53.683+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:53.683+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:53.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:53.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:53.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:53.563+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:54.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:53.614+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:54.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:53.614+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:54.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:53.614+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:54.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:53.615+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:54.703 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:54.507+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23910 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:54.703 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:54.572+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:54.703 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:54.572+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:54.703 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:54.572+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:54.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:54.572+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:54.704 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:54.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:54.704 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:54.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:54.704 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:54.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:54.704 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:54.583+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:54.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:54.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:54.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:54.702+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:55.993 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:55.682+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:55.994 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:55.682+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:55.994 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:55.682+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:55.994 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:55.683+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:55.994 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:55.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:55.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:55.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:55.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:55.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:55.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:55.604+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:55.995 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:55.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:55.995 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:55.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:55.995 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:55.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:55.996 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:55.561+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:56.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:56.576+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:56.576+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:56.576+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:56.577+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:56.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:56.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:56.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:56.527+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:56.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:56.528+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:56.706+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:56.706+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:56.706+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:56.708+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:57.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:57.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:57.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:57.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:57.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:57.578+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:57.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:57.578+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:57.749 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:57.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:57.749 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:57.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:57.749 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:57.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:57.749 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:57.488+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:58.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:57.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:58.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:57.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:57.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:57.747+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:58.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:58.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:58.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:58.503+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:58.505+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:04:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:58.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:58.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:58.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:58.584+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:58.724+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:58.724+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:58.724+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:58.725+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:59.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:04:59.509+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 23962 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:04:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:59.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:59.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:59.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:04:59.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:04:59.631+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:59.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:59.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:59.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:59.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:59.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:04:59.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:04:59.688+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:04:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:59.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:59.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:59.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:04:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:04:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:04:59.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:00.727+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:00.727+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:00.727+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:00.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:00.728+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:00.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:00.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:00.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:00.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:00.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:00.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:00.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:00.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:00.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:00.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:00.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:00.617+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:01.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:01.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:01.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:01.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:01.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:01.482+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:01.751 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:01.483+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:01.751 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:01.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:01.751 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:01.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:01.752 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:01.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:01.752 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:01.640+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:02.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:01.748+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:01.748+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:01.748+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:01.749+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:02.752 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:02.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:02.752 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:02.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:02.752 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:02.459+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:02.752 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:02.459+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:02.752 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:02.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:02.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:02.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:02.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:02.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:02.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:02.685+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:03.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:02.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:03.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:02.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:02.749+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:02.750+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:03.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:03.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:03.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:03.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:03.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:03.643+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:03.753 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:03.644+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:03.754 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:03.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:03.754 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:03.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:03.754 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:03.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:03.754 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:03.427+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:04.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:03.751+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:03.751+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:03.751+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:03.751+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:04.800 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:04.512+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24015 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:04.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:04.677+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:04.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:04.677+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:04.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:04.677+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:04.801 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:04.678+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:04.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:04.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:04.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:04.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:04.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:04.453+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:04.802 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:04.454+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:05.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:04.798+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:04.798+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:04.798+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:05.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:04.799+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:05.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:05.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:05.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:05.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:05.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:05.425+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:05.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:05.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:05.658+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:06.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:05.658+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:05.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:05.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:05.830+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:05.831+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:06.798 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:06.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:06.799 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:06.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:06.799 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:06.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:06.799 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:06.654+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:06.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:06.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:06.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:06.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:06.799 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:06.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:06.800 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:06.424+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:07.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:06.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:06.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:06.796+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:06.797+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:07.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:07.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:07.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:07.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:07.808 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:07.628+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:07.809 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:07.629+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:07.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:07.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:07.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:07.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:07.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:07.409+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:07.809 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:07.410+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:08.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:07.805+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:07.805+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:07.805+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:07.806+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:08.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:08.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:08.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:08.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:08.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:08.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:08.766 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:08.617+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:08.766 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:08.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:08.767 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:08.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:08.767 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:08.369+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:08.767 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:08.370+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:09.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:08.763+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:08.763+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:08.763+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:09.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:08.764+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:09.755 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:09.514+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24066 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:09.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:09.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:09.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:09.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:09.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:09.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:09.756 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:09.411+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:09.756 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:09.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:09.756 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:09.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:09.756 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:09.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:09.757 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:09.591+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:10.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:09.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:10.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:09.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:09.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:09.754+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:10.730 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:10.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:10.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:10.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:10.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:10.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:10.731 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:10.617+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:10.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:10.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:10.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:10.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:10.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:10.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:10.732 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:10.374+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:11.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:10.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:10.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:10.728+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:10.729+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:11.748 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:11.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:11.749 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:11.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:11.749 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:11.376+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:11.749 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:11.377+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:11.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:11.569+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:11.749 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:11.569+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:11.750 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:11.569+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:11.750 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:11.569+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:12.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:11.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:11.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:11.746+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:11.747+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:12.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:12.379+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:12.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:12.379+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:12.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:12.379+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:12.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:12.379+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:12.770 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:12.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:12.770 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:12.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:12.770 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:12.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:12.771 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:12.600+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:13.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:12.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:13.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:12.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:12.767+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:12.768+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:13.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:13.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:13.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:13.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:13.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:13.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:13.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:13.404+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:13.675 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:13.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:13.675 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:13.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:13.675 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:13.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:13.675 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:13.636+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:14.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:13.798+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:13.798+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:13.798+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:13.799+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:14.651 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:14.516+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24119 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:14.651 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:14.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:14.651 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:14.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:14.651 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:14.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:14.652 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:14.364+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:15.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:14.648+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:14.648+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:14.648+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:14.649+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:14.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:14.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:14.753+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:14.754+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:15.715 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:15.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:15.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:15.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:15.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:15.613+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:15.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:15.614+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:15.716 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:15.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:15.716 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:15.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:15.716 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:15.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:15.717 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:15.412+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:16.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:15.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:15.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:15.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:16.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:15.714+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:16.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:16.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:16.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:16.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:16.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:16.454+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:16.731 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:16.455+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:16.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:16.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:16.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:16.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:16.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:16.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:16.732 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:16.620+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:17.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:16.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:16.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:16.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:16.730+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:17.736 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:17.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:17.736 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:17.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:17.737 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:17.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:17.737 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:17.596+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:17.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:17.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:17.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:17.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:17.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:17.458+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:17.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:17.459+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:18.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:17.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:18.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:17.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:17.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:17.735+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:18.784 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:18.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:18.784 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:18.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:18.784 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:18.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:18.784 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:18.550+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:18.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:18.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:18.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:18.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:18.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:18.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:18.785 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:18.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:19.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:18.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:18.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:18.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:19.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:18.783+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:19.746 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:19.519+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24171 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:19.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:19.472+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:19.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:19.472+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:19.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:19.472+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:19.747 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:19.473+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:19.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:19.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:19.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:19.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:19.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:19.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:19.748 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:19.526+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:20.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:19.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:19.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:19.744+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:20.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:19.745+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:20.789 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:20.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:20.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:20.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:20.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:20.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:20.790 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:20.511+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:20.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:20.554+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:20.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:20.554+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:20.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:20.554+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:20.791 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:20.555+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:21.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:20.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:20.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:20.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:20.788+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:21.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:21.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:21.526+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:21.527+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:21.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:21.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:21.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:21.541+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:22.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:21.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:21.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:21.836+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:22.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:21.837+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:22.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:22.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:22.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:22.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:22.537+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:22.530+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:22.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:22.531+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:22.532+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:23.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:22.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:22.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:22.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:23.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:22.840+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:23.533+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:23.533+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:23.533+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:23.534+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:23.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:23.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:23.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:23.490+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:23.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:24.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:23.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:23.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:23.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:24.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:23.819+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:24.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:24.521+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24224 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:24.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:24.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:24.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:24.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:24.550+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:24.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:24.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:24.512+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:24.513+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:25.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:24.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:24.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:24.833+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:25.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:24.834+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:25.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:25.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:25.549+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:25.550+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:25.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:25.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:25.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:25.540+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:25.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:25.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:25.828+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:26.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:25.829+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:26.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:26.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:26.577+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:26.577+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:26.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:26.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:26.797+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:27.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:26.797+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:26.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:26.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:26.567+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:27.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:26.567+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:27.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:27.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:27.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:27.537+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:27.538+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:27.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:27.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:27.596+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:27.597+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:28.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:27.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:27.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:27.835+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:28.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:27.836+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:28.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:28.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:28.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:28.520+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:28.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:28.520+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:28.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:28.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:28.565+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:28.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:28.565+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:28.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:28.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:28.879+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:29.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:28.880+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:29.523+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24275 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:29.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:29.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:29.507+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:29.507+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:29.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:29.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:29.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:29.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:29.571+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:30.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:29.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:29.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:29.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:30.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:29.878+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:30.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:30.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:30.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:30.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:30.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:30.537+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:30.540+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:30.540+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:30.540+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:30.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:30.540+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:31.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:30.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:30.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:30.898+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:31.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:30.899+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:31.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:31.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:31.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:31.492+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:31.493+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:31.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:31.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:31.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:31.490+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:32.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:31.870+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:31.870+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:31.870+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:32.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:31.870+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:32.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:32.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:32.451+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:32.451+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:32.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:32.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:32.448+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:32.449+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:33.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:32.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:32.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:32.867+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:33.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:32.867+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:33.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:33.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:33.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:33.470+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:33.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:33.471+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:33.455+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:33.455+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:33.455+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:33.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:33.456+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:34.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:33.889+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:33.889+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:33.889+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:34.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:33.891+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:34.736 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:34.526+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24328 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:34.736 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:34.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:34.736 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:34.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:34.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:34.426+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:34.737 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:34.427+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:34.737 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:34.468+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:34.737 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:34.468+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:34.737 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:34.468+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:34.737 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:34.469+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:35.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:34.914+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:34.914+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:34.914+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:34.915+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:35.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:35.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:35.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:35.466+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:35.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:35.467+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:35.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:35.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:35.498+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:35.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:35.499+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:36.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:35.878+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:35.878+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:35.878+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:36.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:35.878+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:36.530+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:36.530+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:36.530+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:36.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:36.531+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:36.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:36.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:36.441+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:36.442+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:36.912+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:36.912+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:36.912+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:37.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:36.913+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:37.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:37.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:37.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:37.506+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:37.507+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:37.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:37.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:37.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:37.479+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:38.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:37.884+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:37.884+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:37.884+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:38.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:37.885+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:38.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:38.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:38.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:38.549+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:38.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:38.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:38.476+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:38.477+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:39.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:38.889+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:38.889+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:38.889+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:39.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:38.889+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:39.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:39.528+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24380 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:39.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:39.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:39.524+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:39.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:39.524+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:39.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:39.523+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:39.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:39.523+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:39.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:39.523+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:39.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:39.524+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:40.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:39.916+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:39.916+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:39.916+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:40.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:39.917+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:40.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:40.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:40.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:40.504+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:40.505+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:40.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:40.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:40.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:40.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:40.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:40.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:40.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:40.536+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:41.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:40.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:40.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:40.877+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:41.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:40.878+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:41.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:41.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:41.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:41.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:41.520+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:41.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:41.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:41.541+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:41.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:41.541+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:41.832+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:41.832+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:41.832+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:42.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:41.833+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:42.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:42.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:42.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:42.471+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:42.472+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:42.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:42.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:42.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:42.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:42.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:42.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:42.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:42.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:42.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:42.866+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:43.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:42.867+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:43.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:43.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:43.489+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:43.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:43.490+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:43.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:43.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:43.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:43.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:43.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:43.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:43.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:43.583+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:44.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:43.910+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:43.910+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:43.910+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:44.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:43.911+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:44.631 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:44.530+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24433 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:44.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:44.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:44.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:44.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:44.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:44.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:44.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:44.535+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:44.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:44.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:44.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:44.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:44.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:44.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:44.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:44.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:45.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:44.934+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:44.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:44.935+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:45.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:44.935+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:45.597+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:45.597+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:45.597+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:45.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:45.598+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:45.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:45.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:45.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:45.488+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:45.489+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:46.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:45.892+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:45.892+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:45.892+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:46.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:45.893+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:46.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:46.555+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:46.555+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:46.555+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:46.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:46.556+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:46.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:46.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:46.446+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:46.446+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:47.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:46.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:46.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:46.905+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:47.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:46.906+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:47.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:47.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:47.704 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:47.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:47.704 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:47.436+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:47.704 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:47.437+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:47.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:47.510+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:47.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:47.510+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:47.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:47.510+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:47.705 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:47.510+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:48.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:47.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:47.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:47.857+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:47.858+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:48.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:48.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:48.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:48.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:48.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:48.490+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:48.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:48.490+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:48.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:48.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:48.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:48.420+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:48.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:48.421+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:49.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:48.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:48.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:48.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:49.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:48.818+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:49.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:49.533+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24484 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:49.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:49.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:49.486+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:49.488+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:49.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:49.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:49.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:49.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:49.398+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:50.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:49.865+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:49.865+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:49.865+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:49.866+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:50.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:50.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:50.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:50.534+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:50.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:50.535+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:50.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:50.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:50.435+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:50.436+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:51.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:50.887+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:50.887+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:50.887+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:51.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:50.888+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:51.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:51.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:51.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:51.497+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:51.498+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:51.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:51.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:51.473+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:51.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:51.474+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:52.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:51.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:51.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:51.881+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:52.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:51.882+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:52.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:52.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:52.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:52.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:52.485+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:52.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:52.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:52.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:52.505+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:52.506+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:53.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:52.904+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:52.904+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:52.904+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:53.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:52.905+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:53.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:53.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:53.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:53.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:53.548+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:53.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:53.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:53.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:53.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:53.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:53.508+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:53.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:53.508+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:54.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:53.933+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:53.933+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:53.933+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:54.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:53.934+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:54.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:54.535+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24537 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:54.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:54.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:54.538+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:54.539+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:54.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:54.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:54.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:54.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:54.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:54.480+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:54.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:54.481+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:55.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:54.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:54.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:54.931+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:55.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:54.932+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:55.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:55.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:55.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:55.494+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:55.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:55.495+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:55.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:55.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:55.548+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:55.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:55.548+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:56.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:55.947+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:55.947+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:55.947+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:56.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:55.948+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:56.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:56.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:56.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:56.518+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:56.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:56.519+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:56.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:56.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:56.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:56.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:56.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:56.557+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:57.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:56.955+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:57.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:56.955+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:57.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:56.955+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:57.249 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:56.955+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:57.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:57.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:57.606+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:57.606+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:57.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:57.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:57.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:57.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:57.505+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:58.184 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:57.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:58.184 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:57.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:58.184 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:57.980+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:58.184 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:57.981+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:58.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:58.533+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:58.533+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:58.533+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:58.533+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:58.578+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:58.578+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:58.578+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:58.579+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:59.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:58.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:58.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:58.999+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:05:59.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:05:59.000+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:05:59.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:05:59.537+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24588 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:05:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:59.612+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:59.612+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:59.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:59.612+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:05:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:05:59.613+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:05:59.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:59.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:59.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:59.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:59.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:59.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:05:59.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:05:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:05:59.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:00.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.003+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:00.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.004+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:00.981 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:00.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:00.982 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:00.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:00.982 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:00.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:00.982 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:00.598+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:00.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:00.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:00.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:00.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:00.982 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:00.608+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:00.983 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:00.609+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:01.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.979+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:01.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:00.980+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:01.951 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:01.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:01.951 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:01.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:01.952 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:01.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:01.952 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:01.658+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:01.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:01.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:01.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:01.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:01.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:01.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:01.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:01.564+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:02.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:01.949+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:01.949+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:01.949+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:02.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:01.950+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:02.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:02.586+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:02.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:02.586+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:02.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:02.586+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:02.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:02.587+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:02.947 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:02.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:02.947 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:02.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:02.947 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:02.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:02.947 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:02.657+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:03.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:02.944+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:02.944+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:02.944+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:03.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:02.945+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:03.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:03.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:03.914 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:03.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:03.914 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:03.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:03.914 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:03.607+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:03.914 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:03.619+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:03.915 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:03.619+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:03.915 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:03.619+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:03.915 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:03.620+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:04.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:03.911+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:03.911+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:03.911+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:04.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:03.912+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:04.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:04.540+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24640 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:04.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:04.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:04.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:04.577+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:04.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:04.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:04.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:04.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:04.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:04.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:04.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:04.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:05.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:04.896+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:04.896+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:04.896+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:04.897+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:05.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:05.566+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:05.566+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:05.566+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:05.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:05.568+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:05.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:05.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:05.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:05.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:05.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:05.528+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:06.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:05.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:05.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:05.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:05.864+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:06.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:06.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:06.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:06.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:06.537+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:06.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:06.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:06.585+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:06.586+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:07.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:06.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:06.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:06.816+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:06.817+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:08.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:07.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:07.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:07.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:07.583+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:07.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:07.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:07.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:07.788+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:07.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:07.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:07.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:07.610+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:08.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:08.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:08.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:08.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:08.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:08.785+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:08.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:08.786+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:08.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:08.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:08.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:08.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:08.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:08.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:08.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:08.589+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:08.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:08.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:08.564+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:08.565+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:09.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:09.542+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24693 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:09.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:09.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:09.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:09.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:09.588+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:09.561+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:09.561+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:09.561+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:09.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:09.562+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:09.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:09.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:09.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:10.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:09.814+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:11.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:10.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:10.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:10.800+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:11.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:10.801+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:10.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:10.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:10.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:11.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:10.588+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:10.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:11.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:10.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:11.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:10.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:11.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:10.574+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:12.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:11.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:11.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:11.601+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:11.602+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:12.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:11.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:11.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:11.781+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:11.782+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:12.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:11.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:12.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:11.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:12.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:11.575+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:12.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:11.576+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:13.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:12.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:12.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:12.812+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:12.813+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:12.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:12.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:12.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:12.574+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:12.579+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:12.579+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:13.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:12.579+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:13.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:12.580+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:14.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:13.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:13.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:13.768+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:13.769+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:13.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:13.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:13.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:14.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:13.560+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:13.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:13.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:13.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:14.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:13.589+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:14.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:14.544+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24745 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:14.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:14.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:14.616+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:14.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:14.617+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:14.766+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:14.766+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:14.766+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:14.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:14.767+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:14.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:14.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:14.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:14.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:14.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:14.559+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:15.829 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:15.741+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:15.829 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:15.741+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:15.829 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:15.741+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:15.830 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:15.741+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:15.830 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:15.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:15.830 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:15.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:15.830 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:15.570+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:15.830 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:15.571+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:15.831 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:15.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:15.831 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:15.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:15.831 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:15.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:15.831 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:15.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:16.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:16.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:16.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:16.527+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:16.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:16.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:16.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:16.584+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:16.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:16.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:16.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:16.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:16.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:16.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:16.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:16.715+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:17.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:17.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:17.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:17.599+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:17.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:17.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:17.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:17.667+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:17.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:17.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:17.532+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:17.533+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:18.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:18.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:18.637+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:18.637+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:18.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:18.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:18.530+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:18.531+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:18.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:18.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:18.559+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:18.559+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:19.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:19.547+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24798 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:19.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:19.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:19.567+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:19.568+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:19.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:19.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:19.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:19.640+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:19.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:19.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:19.550+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:19.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:19.551+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:20.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:20.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:20.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:20.570+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:20.571+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:20.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:20.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:20.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:20.641+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:20.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:20.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:20.526+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:20.527+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:21.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:21.682+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:21.682+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:21.682+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:21.682+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:21.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:21.501+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:21.501+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:21.501+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:21.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:21.502+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:21.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:21.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:21.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:21.604+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:22.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:22.591+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:22.591+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:22.591+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:22.592+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:22.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:22.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:22.713+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:22.714+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:22.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:22.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:22.520+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:22.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:22.520+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:23.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:23.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:23.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:23.677+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:23.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:23.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:23.519+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:23.520+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:23.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:23.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:23.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:23.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:23.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:23.609+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:23.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:23.610+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:24.728 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:24.549+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24850 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:24.728 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:24.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:24.729 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:24.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:24.729 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:24.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:24.729 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:24.475+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:24.729 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:24.476+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:24.729 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:24.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:24.729 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:24.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:24.730 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:24.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:24.730 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:24.608+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:25.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:24.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:24.725+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:25.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:24.726+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:25.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:25.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:25.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:25.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:25.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:25.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:25.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:25.438+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:25.703 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:25.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:25.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:25.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:25.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:25.600+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:25.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:25.601+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:26.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:25.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:25.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:25.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:25.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:26.701 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:26.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:26.701 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:26.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:26.701 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:26.599+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:26.702 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:26.599+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:26.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:26.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:26.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:26.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:26.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:26.389+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:26.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:26.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:27.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:26.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:26.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:26.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:26.699+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:27.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:27.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:27.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:27.351+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:27.352+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:28.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:27.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:27.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:27.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:27.623+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:27.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:27.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:27.712+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:27.712+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:28.629 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:28.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:28.629 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:28.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:28.630 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:28.365+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:28.630 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:28.366+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:28.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:28.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:28.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:28.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:28.976 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:28.628+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:28.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:28.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:28.718+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:28.719+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:29.673 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:29.551+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24903 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:29.673 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:29.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:29.673 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:29.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:29.673 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:29.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:29.674 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:29.611+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:29.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:29.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:29.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:29.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:29.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:29.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:29.674 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:29.412+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:30.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:29.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:29.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:29.670+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:30.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:29.672+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:30.718 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:30.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:30.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:30.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:30.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:30.445+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:30.719 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:30.446+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:30.719 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:30.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:30.719 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:30.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:30.720 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:30.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:30.720 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:30.639+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:31.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:30.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:30.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:30.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:30.717+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:31.678 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:31.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:31.678 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:31.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:31.678 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:31.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:31.679 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:31.402+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:31.679 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:31.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:31.679 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:31.595+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:31.679 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:31.596+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:31.679 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:31.596+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:32.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:31.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:31.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:31.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:31.677+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:32.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:32.660+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:32.660+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:32.660+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:32.661+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:32.589+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:32.589+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:32.589+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:32.590+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:32.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:32.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:32.438+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:32.439+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:33.679 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:33.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:33.679 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:33.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:33.680 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:33.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:33.680 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:33.543+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:33.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:33.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:33.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:33.680 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:33.650+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:33.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:33.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:33.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:33.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:33.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:33.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:33.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:33.410+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:34.556 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:34.419+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:34.557 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:34.419+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:34.557 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:34.419+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:34.557 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:34.420+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:34.554+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 24955 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:34.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:34.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:34.583+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:34.584+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:34.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:34.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:34.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:34.662+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:35.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:35.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:35.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:35.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:35.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:35.377+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:36.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:35.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:35.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:35.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:36.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:35.637+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:35.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:36.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:35.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:36.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:35.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:36.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:35.610+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:36.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:36.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:36.599+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:36.599+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:36.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:36.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:36.424+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:36.425+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:36.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:36.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:36.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:36.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:36.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:36.653+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:37.465+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:37.465+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:37.465+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:37.465+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:37.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:37.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:37.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:37.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:37.626+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:37.555+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:37.555+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:37.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:37.555+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:37.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:37.556+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:38.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:38.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:38.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:38.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:38.655+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:38.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:38.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:38.532+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:38.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:38.533+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:38.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:38.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:38.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:38.478+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:38.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:38.479+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:39.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:39.556+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25008 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:39.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:39.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:39.670+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:39.670+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:39.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:39.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:39.496+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:39.497+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:39.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:39.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:39.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:39.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:39.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:39.546+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:40.529 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:40.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:40.529 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:40.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:40.529 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:40.469+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:40.529 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:40.470+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:40.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:40.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:40.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:40.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:40.673+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:40.526+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:40.526+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:40.526+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:40.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:40.527+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:41.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:41.427+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:41.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:41.427+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:41.702 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:41.427+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:41.703 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:41.428+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:41.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:41.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:41.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:41.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:41.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:41.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:41.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:41.575+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:41.700+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:41.700+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:41.700+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:41.700+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:42.713 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:42.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:42.713 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:42.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:42.714 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:42.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:42.714 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:42.563+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:42.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:42.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:42.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:42.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:42.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:42.388+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:42.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:42.389+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:42.711+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:42.711+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:42.711+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:42.711+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:43.680 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:43.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:43.680 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:43.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:43.680 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:43.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:43.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:43.570+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:43.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:43.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:43.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:43.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:43.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:43.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:43.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:43.396+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:44.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:43.725+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:43.725+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:43.725+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:43.726+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:44.692 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:44.558+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25060 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:44.692 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:44.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:44.693 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:44.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:44.693 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:44.401+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:44.693 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:44.402+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:44.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:44.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:44.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:44.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:44.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:44.539+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:44.694 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:44.540+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:44.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:44.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:44.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:45.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:44.691+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:45.493+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:45.493+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:45.493+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:45.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:45.494+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:45.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:45.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:45.397+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:45.562 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:45.398+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:46.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:45.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:45.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:45.740+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:45.741+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:46.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:46.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:46.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:46.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:46.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:46.403+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:46.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:46.404+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:46.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:46.492+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:46.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:46.492+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:46.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:46.492+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:46.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:46.494+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:47.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:46.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:46.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:46.759+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:46.760+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:47.743 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:47.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:47.743 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:47.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:47.744 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:47.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:47.744 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:47.456+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:47.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:47.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:47.744 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:47.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:47.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:47.430+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:47.745 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:47.431+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:47.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:47.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:47.741+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:47.741+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:48.771 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:48.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:48.771 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:48.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:48.772 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:48.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:48.772 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:48.454+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:48.772 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:48.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:48.772 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:48.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:48.772 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:48.390+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:48.773 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:48.390+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:48.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:48.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:48.769+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:49.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:48.770+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:49.561+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25113 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:49.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:49.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:49.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:49.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:49.431+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:49.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:49.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:49.363+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:49.364+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:49.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:49.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:49.814+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:49.815+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:50.463+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:50.463+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:50.463+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:50.464+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:50.406+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:50.406+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:50.406+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:50.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:50.407+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:51.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:50.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:50.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:50.864+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:51.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:50.865+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:51.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:51.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:51.458+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:51.459+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:51.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:51.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:51.429+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:51.430+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:52.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:51.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:51.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:51.820+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:51.821+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:52.790 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:52.481+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:52.791 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:52.481+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:52.791 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:52.481+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:52.791 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:52.482+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:52.791 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:52.791 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:52.792 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:52.381+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:52.792 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:52.382+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:53.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:52.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:52.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:52.788+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:53.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:52.789+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:53.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:53.440+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:53.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:53.440+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:53.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:53.440+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:53.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:53.440+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:53.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:53.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:53.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:53.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:53.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:53.337+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:53.682 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:53.338+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:54.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:53.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:53.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:53.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:53.761+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:54.753 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:54.563+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25165 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:54.753 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:54.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:54.753 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:54.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:54.753 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:54.348+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:54.754 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:54.349+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:54.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:54.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:54.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:54.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:54.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:54.418+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:54.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:54.419+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:55.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:54.750+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:54.750+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:54.750+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:55.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:54.751+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:55.777 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:55.438+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:55.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:55.438+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:55.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:55.438+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:55.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:55.440+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:55.778 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:55.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:55.778 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:55.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:55.778 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:55.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:55.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:55.384+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:55.775+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:55.775+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:55.775+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:55.776+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:56.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:56.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:56.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:56.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:56.769 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:56.410+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:56.769 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:56.411+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:56.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:56.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:56.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:56.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:56.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:56.377+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:56.770 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:56.379+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:57.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:56.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:56.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:56.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:57.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:56.767+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:57.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:57.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:57.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:57.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:57.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:57.402+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:57.768 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:57.403+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:57.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:57.379+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:57.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:57.379+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:57.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:57.379+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:57.769 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:57.380+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:58.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:57.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:57.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:57.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:57.767+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:58.761 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:58.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:58.761 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:58.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:58.761 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:58.439+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:58.762 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:58.440+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:58.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:58.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:58.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:58.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:58.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:58.402+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:58.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:58.403+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:06:59.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:58.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:58.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:58.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:06:59.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:58.759+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:59.737 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:06:59.566+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25218 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:06:59.737 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:59.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:59.737 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:59.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:59.737 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:59.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:06:59.738 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:06:59.438+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:06:59.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:59.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:59.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:59.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:59.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:59.364+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:06:59.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:06:59.365+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:00.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:59.734+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:59.734+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:59.734+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:00.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:06:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:06:59.735+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:00.779 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:00.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:00.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:00.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:00.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:00.430+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:00.780 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:00.431+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:00.780 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:00.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:00.780 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:00.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:00.781 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:00.336+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:00.781 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:00.337+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:01.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:00.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:00.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:00.777+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:01.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:00.778+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:01.793 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:01.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:01.793 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:01.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:01.793 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:01.471+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:01.794 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:01.471+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:01.794 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:01.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:01.794 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:01.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:01.794 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:01.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:01.794 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:01.385+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:02.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:01.791+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:01.791+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:01.791+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:02.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:01.792+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:02.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:02.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:02.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:02.373+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:02.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:02.374+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:02.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:02.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:02.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:02.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:02.502+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:02.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:02.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:02.841+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:03.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:02.842+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:03.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:03.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:03.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:03.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:03.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:03.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:03.682 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:03.469+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:03.686 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:03.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:03.686 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:03.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:03.686 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:03.413+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:03.686 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:03.414+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:04.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:03.811+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:03.811+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:03.811+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:04.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:03.812+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:04.777 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:04.568+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25270 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:04.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:04.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:04.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:04.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:04.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:04.422+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:04.778 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:04.422+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:04.778 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:04.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:04.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:04.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:04.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:04.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:04.779 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:04.426+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:05.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:04.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:04.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:04.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:05.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:04.775+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:05.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:05.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:05.451+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:05.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:05.452+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:05.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:05.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:05.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:05.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:05.411+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:05.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:05.412+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:06.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:05.811+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:05.811+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:05.811+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:06.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:05.812+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:06.419+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:06.419+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:06.419+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:06.420+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:06.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:06.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:06.425+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:06.425+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:06.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:06.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:06.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:06.843+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:07.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:07.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:07.382+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:07.383+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:07.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:07.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:07.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:07.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:07.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:07.449+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:07.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:07.450+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:08.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:07.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:07.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:07.831+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:08.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:07.831+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:08.792 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:08.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:08.792 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:08.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:08.793 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:08.414+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:08.793 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:08.415+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:08.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:08.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:08.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:08.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:08.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:08.384+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:38.249551+0000 (oldest deadline 2024-03-12T03:56:58.249551+0000) 2024-03-12T04:07:08.793 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:08.385+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:09.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:08.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:09.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:08.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:09.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:08.790+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:09.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:08.791+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:09.797 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:09.570+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25323 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:09.797 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:09.357+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:09.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:09.373+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:09.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:09.373+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:09.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:09.373+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:09.798 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:09.373+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:10.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:09.794+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:09.794+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:09.794+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:09.795+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:10.762 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:10.385+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:10.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:10.394+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:10.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:10.394+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:10.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:10.394+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:10.763 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:10.395+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:11.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:10.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:10.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:10.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:11.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:10.760+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:11.738 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:11.341+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:11.738 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:11.373+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:11.738 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:11.373+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:11.738 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:11.373+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:11.738 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:11.374+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:11.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:11.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:11.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:12.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:11.736+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:12.695 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:12.308+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:12.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:12.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:12.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:12.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:12.696 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:12.385+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:12.696 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:12.386+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:12.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:12.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:12.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:12.694+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:13.680 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:13.342+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:13.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:13.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:13.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:13.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:13.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:13.362+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:13.681 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:13.363+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:14.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:13.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:13.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:13.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:14.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:13.679+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:14.386 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:14.294+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:14.648 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:14.572+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25375 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:14.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:14.384+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:14.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:14.384+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:14.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:14.384+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:14.648 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:14.385+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:14.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:14.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:14.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:14.646+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:15.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:15.288+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:15.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:15.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:15.380+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:15.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:15.381+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:16.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:15.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:15.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:15.669+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:15.670+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:16.673 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:16.315+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:16.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:16.426+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:16.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:16.426+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:16.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:16.426+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:16.674 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:16.427+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:17.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:16.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:16.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:16.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:17.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:16.672+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:17.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:17.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:17.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:17.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:17.641 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:17.401+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:17.642 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:17.402+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:17.642 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:17.324+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:18.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:17.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:17.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:17.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:18.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:17.640+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:18.635 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:18.370+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:18.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:18.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:18.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:18.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:18.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:18.357+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:18.636 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:18.358+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:18.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:18.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:18.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:18.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:18.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:18.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:18.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:18.634+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:19.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:19.575+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25428 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:19.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:19.374+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:19.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:19.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:19.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:19.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:19.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:19.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:19.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:19.612+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:19.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:19.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:19.389+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:19.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:19.390+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:20.612 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:20.355+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:20.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:20.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:20.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:20.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:20.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:20.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:20.613 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:20.417+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:21.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:20.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:20.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:20.610+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:21.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:20.611+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:21.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:21.307+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:21.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:21.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:21.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:21.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:21.456+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:22.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:21.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:21.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:21.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:52.271088+0000 (oldest deadline 2024-03-12T03:57:12.271088+0000) 2024-03-12T04:07:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:21.618+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:22.639 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:22.332+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:22.639 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:22.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:22.639 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:22.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:22.639 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:22.443+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T03:56:47.299347+0000 (oldest deadline 2024-03-12T03:57:07.299347+0000) 2024-03-12T04:07:22.639 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:22.444+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:23.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:22.636+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:23.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:23.482+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:23.653 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:23.377+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:24.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:23.650+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:24.646 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:24.577+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25479 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:24.647 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:24.478+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:24.647 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:24.338+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:24.644+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:25.634 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:25.441+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:25.635 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:25.367+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:26.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:25.632+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:26.615 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:26.481+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:26.616 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:26.327+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:26.616 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:26.613+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:27.606 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:27.309+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:27.607 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:27.477+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:28.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:27.604+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:28.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:28.592+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:28.503+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:28.344+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:29.580+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25531 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:29.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:29.586+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:29.553+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:29.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:29.353+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:30.623 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:30.537+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:30.623 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:30.550+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:30.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:30.324+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:30.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:30.324+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:30.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:30.324+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:30.624 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:30.325+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:31.515+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:31.535+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:31.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:31.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:31.375+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:31.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:31.375+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:32.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:32.529+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:32.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:32.554+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:32.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:32.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:32.395+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:32.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:32.396+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:33.683 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:33.557+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:33.684 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:33.522+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:33.684 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:33.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:33.684 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:33.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:33.685 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:33.407+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:33.685 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:33.408+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:34.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:34.582+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25583 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:34.520+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:34.540+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:34.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:34.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:34.383+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:34.384+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:35.667 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:35.550+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:35.667 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:35.532+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:35.667 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:35.385+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:35.668 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:35.385+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:35.668 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:35.385+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:35.668 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:35.386+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:36.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:36.535+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:36.545+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:36.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:36.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:36.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:36.410+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:36.410+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:37.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:37.511+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:37.497+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:37.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:37.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:37.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:37.423+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:37.424+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:38.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:38.496+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:38.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:38.467+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:38.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:38.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:38.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:38.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:38.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:38.450+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:39.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:39.504+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:39.422+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:39.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:39.585+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25636 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:39.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:39.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:39.428+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:39.429+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:40.709 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:40.390+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:40.710 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:40.533+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:40.710 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:40.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:40.710 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:40.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:40.710 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:40.437+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:40.711 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:40.437+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:41.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:41.573+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:41.382+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:41.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:41.462+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:41.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:41.463+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:42.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:42.416+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:42.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:42.572+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:42.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:42.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:42.456+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:42.457+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:43.685 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:43.596+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:43.685 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:43.416+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:43.685 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:43.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:43.686 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:43.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:43.686 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:43.448+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:43.686 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:43.448+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:44.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:44.587+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25688 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:44.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:44.382+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:44.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:44.616+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:44.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:44.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:44.463+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:44.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:44.464+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:45.631 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:45.352+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:45.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:45.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:45.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:45.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:45.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:45.461+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:45.632 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:45.461+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:45.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:45.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:45.629+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:45.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:46.374+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:46.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:46.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:46.479+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:46.480+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:46.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:46.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:46.603+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:46.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:46.604+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:47.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:47.389+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:47.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:47.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:47.510+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:47.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:47.512+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:47.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:47.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:47.568+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:47.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:47.568+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:48.394+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:48.394+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:48.394+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:48.394+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:48.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:48.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:48.524+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:48.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:48.525+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:48.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:48.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:48.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:48.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:48.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:48.547+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:48.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:48.547+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:49.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:49.590+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25741 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:49.476+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:49.476+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:49.476+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:49.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:49.477+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:49.436+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:49.436+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:49.436+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:49.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:49.436+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:49.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:49.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:49.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:49.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:49.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:49.587+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:49.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:49.588+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:50.789 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:50.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:50.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:50.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:50.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:50.445+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:50.790 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:50.445+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:50.790 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:50.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:50.790 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:50.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:50.791 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:50.446+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:50.791 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:50.447+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:50.791 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:50.561+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:50.791 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:50.561+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:50.791 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:50.561+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:50.791 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:50.562+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:51.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:51.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:51.417+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:51.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:51.418+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:51.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:51.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:51.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:51.534+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:51.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:51.534+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:51.442+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:51.442+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:51.442+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:51.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:51.443+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:52.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:52.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:52.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:52.462+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:52.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:52.463+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:52.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:52.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:52.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:52.382+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:52.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:52.382+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:52.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:52.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:52.574+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:52.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:52.575+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:53.685 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:53.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:53.685 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:53.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:53.685 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:53.340+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:53.686 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:53.341+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:53.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:53.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:53.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:53.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:53.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:53.501+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:53.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:53.502+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:53.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:53.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:53.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:53.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:53.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:53.575+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:53.687 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:53.576+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:54.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:54.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:54.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:54.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:54.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:54.324+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:54.491 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:54.325+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:54.492 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:54.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:54.492 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:54.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:54.492 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:54.488+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:54.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:54.592+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25793 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:07:54.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:54.489+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:54.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:54.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:54.535+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:54.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:54.535+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:55.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:55.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:55.350+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:55.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:55.351+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:55.503+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:55.503+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:55.503+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:55.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:55.504+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:55.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:55.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:55.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:55.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:55.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:55.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:55.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:55.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:56.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:56.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:56.539+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:56.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:56.540+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:56.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:56.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:56.553+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:56.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:56.554+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:56.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:56.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:56.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:56.308+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:56.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:56.309+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:57.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:57.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:57.546+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:57.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:57.547+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:57.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:57.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:57.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:57.297+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:57.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:57.298+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:58.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:57.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:57.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:57.563+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:58.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:57.564+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:58.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:58.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:58.333+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:58.335+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:58.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:58.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:58.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:58.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:58.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:58.551+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:58.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:58.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:58.566+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:58.566+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:59.597 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:59.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:59.597 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:59.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:59.598 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:59.327+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:07:59.598 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:07:59.328+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:07:59.598 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:59.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:59.598 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:59.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:59.598 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:59.529+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:07:59.598 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:07:59.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:07:59.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:59.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:59.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:59.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:59.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:59.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:07:59.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:07:59.539+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:00.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:07:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:07:59.595+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25846 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:00.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:00.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:00.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:00.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:00.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:00.296+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:00.554 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:00.297+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:00.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:00.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:00.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:00.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:00.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:00.538+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:00.555 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:00.539+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:00.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:00.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:00.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:00.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:00.551+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:00.551+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:01.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:01.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:01.495+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:01.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:01.496+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:01.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:01.320+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:01.320+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:01.320+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:01.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:01.321+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:01.542+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:01.542+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:01.542+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:01.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:01.543+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:02.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:02.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:02.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:02.537+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:02.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:02.538+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:02.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:02.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:02.504+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:02.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:02.505+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:02.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:02.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:02.293+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:02.562 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:02.294+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:03.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:03.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:03.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:03.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:03.686 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:03.519+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:03.687 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:03.520+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:03.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:03.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:03.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:03.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:03.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:03.341+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:03.687 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:03.342+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:03.688 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:03.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:03.688 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:03.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:03.688 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:03.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:03.688 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:03.580+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:04.599 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:04.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:04.600 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:04.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:04.600 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:04.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:04.600 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:04.543+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:04.600 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:04.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:04.600 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:04.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:04.600 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:04.309+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:04.600 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:04.310+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:05.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:04.597+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25898 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:04.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:04.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:04.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:04.619+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:05.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:05.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:05.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:05.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:05.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:05.345+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:05.626 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:05.346+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:05.626 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:05.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:05.627 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:05.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:05.627 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:05.550+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:05.627 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:05.551+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:05.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:05.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:05.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:05.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:05.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:05.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:05.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:05.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:06.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:06.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:06.367+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:06.367+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:06.598+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:06.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:06.598+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:06.598+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:06.599+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:06.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:06.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:06.548+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:06.549+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:07.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:07.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:07.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:07.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:07.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:07.332+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:07.619 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:07.333+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:07.619 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:07.552+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:07.619 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:07.552+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:07.620 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:07.552+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:07.620 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:07.553+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:08.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:07.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:07.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:07.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:07.617+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:08.652 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:08.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:08.652 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:08.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:08.652 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:08.536+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:08.652 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:08.537+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:08.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:08.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:08.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:08.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:08.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:08.354+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:08.653 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:08.356+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:08.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:08.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:08.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:08.651+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:09.694 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:09.600+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 25951 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:09.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:09.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:09.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:09.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:09.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:09.391+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:09.695 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:09.392+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:09.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:09.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:09.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:09.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:09.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:09.542+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:09.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:09.543+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:09.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:09.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:09.692+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:10.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:09.693+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:10.689 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:10.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:10.689 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:10.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:10.690 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:10.425+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:10.690 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:10.426+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:10.690 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:10.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:10.690 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:10.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:10.690 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:10.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:10.690 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:10.587+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:10.948 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:10.687+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:10.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:10.687+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:10.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:10.687+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:10.949 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:10.688+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:11.723 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:11.561+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:11.723 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:11.561+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:11.723 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:11.561+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:11.723 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:11.561+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:11.723 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:11.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:11.724 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:11.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:11.724 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:11.453+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:11.724 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:11.453+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:12.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:11.720+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:11.720+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:11.720+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:11.721+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:12.431+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:12.431+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:12.431+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:12.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:12.433+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:12.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:12.674+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:12.674+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:12.674+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:12.675+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:12.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:12.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:12.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:12.607+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:12.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:12.608+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:13.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:13.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:13.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:13.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:13.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:13.437+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:13.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:13.438+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:13.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:13.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:13.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:13.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:13.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:13.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:13.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:13.618+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:14.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:13.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:13.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:13.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:13.713+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:14.753 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:14.602+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26003 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:14.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:14.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:14.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:14.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:14.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:14.455+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:14.754 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:14.456+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:14.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:14.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:14.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:14.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:14.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:14.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:14.755 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:14.647+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:15.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:14.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:14.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:14.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:14.752+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:15.750 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:15.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:15.751 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:15.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:15.751 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:15.482+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:15.751 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:15.483+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:15.751 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:15.679+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:15.751 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:15.679+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:15.752 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:15.679+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:15.752 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:15.680+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:16.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:15.748+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:15.748+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:15.748+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:15.749+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:16.720+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:16.720+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:16.720+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:16.721+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:16.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:16.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:16.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:16.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:16.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:16.494+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:16.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:16.495+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:16.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:16.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:16.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:16.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:16.707+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:17.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:17.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:17.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:17.748+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:17.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:17.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:17.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:17.524+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:17.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:17.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:17.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:17.685+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:18.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:18.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:18.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:18.735+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:18.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:18.736+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:18.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:18.704+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:18.704+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:18.704+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:18.704+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:18.542+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:18.542+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:18.542+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:18.542+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:20.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:19.604+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26056 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:19.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:19.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:19.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:20.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:19.656+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:20.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:19.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:20.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:19.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:20.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:19.749+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:20.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:19.750+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:20.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:19.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:20.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:19.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:20.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:19.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:20.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:19.566+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:20.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:20.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:20.784+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:20.784+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:20.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:20.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:20.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:20.623+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:20.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:20.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:20.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:20.550+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:20.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:20.552+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:22.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:21.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:21.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:21.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:21.648+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:21.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:21.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:22.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:21.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:22.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:21.748+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:21.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:21.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:21.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:21.568+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:22.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:22.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:22.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:22.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:22.682+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:22.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:22.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:22.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:22.766+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:22.767+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:22.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:22.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:22.548+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:22.548+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:24.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:23.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:23.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:23.706+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:23.707+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:23.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:23.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:24.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:23.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:24.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:23.756+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:23.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:23.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:23.559+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:23.560+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:24.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:24.607+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26108 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:24.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:24.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:24.516+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:24.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:24.516+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:24.764+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:24.764+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:24.764+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:24.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:24.765+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:24.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:24.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:24.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:24.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:24.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:24.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:24.673+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:25.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:25.716+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:25.716+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:25.716+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:25.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:25.717+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:25.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:25.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:25.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:25.683+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:25.684+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:25.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:25.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:25.501+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:25.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:25.502+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:26.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:26.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:26.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:26.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:26.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:26.712+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:26.757 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:26.713+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:26.758 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:26.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:26.758 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:26.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:26.758 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:26.490+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:26.758 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:26.491+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:27.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:26.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:26.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:26.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:27.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:26.756+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:27.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:27.444+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:27.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:27.444+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:27.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:27.444+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:27.745 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:27.445+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:27.746 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:27.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:27.746 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:27.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:27.746 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:27.672+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:27.746 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:27.673+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:28.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:27.742+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:27.742+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:27.742+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:28.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:27.743+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:28.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:28.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:28.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:28.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:28.686+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:28.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:28.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:28.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:28.469+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:28.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:28.470+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:28.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:28.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:28.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:28.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:28.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:28.715+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:29.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:29.609+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26161 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:29.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:29.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:29.470+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:29.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:29.471+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:29.672+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:29.672+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:29.672+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:29.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:29.673+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:29.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:29.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:29.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:29.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:29.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:29.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:29.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:29.666+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:30.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:30.624+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:30.624+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:30.624+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:30.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:30.625+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:30.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:30.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:30.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:30.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:30.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:30.498+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:30.624+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:30.624+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:30.624+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:30.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:30.625+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:31.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:31.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:31.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:31.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:31.666+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:31.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:31.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:31.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:31.588+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:31.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:31.589+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:31.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:31.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:31.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:31.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:31.524+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:32.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:32.708+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:32.708+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:32.708+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:32.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:32.709+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:32.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:32.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:32.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:32.587+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:32.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:32.588+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:32.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:32.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:32.484+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:32.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:32.485+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:33.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:33.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:33.627+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:33.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:33.628+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:33.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:33.480+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:33.480+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:33.480+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:33.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:33.481+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:33.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:33.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:33.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:33.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:33.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:33.690+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:34.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:34.612+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26213 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:34.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:34.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:34.617+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:34.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:34.618+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:34.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:34.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:34.502+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:34.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:34.503+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:34.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:34.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:34.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:34.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:34.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:34.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:34.736+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:35.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:35.647+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:35.647+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:35.647+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:35.648+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:35.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:35.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:35.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:35.514+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:35.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:35.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:35.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:35.725+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:36.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:36.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:36.514+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:36.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:36.515+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:36.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:36.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:36.663+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:36.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:36.663+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:36.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:36.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:36.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:36.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:36.744+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:36.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:36.745+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:37.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:37.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:37.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:37.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:37.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:37.700+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:37.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:37.519+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:37.519+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:37.519+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:37.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:37.520+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:37.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:37.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:37.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:37.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:37.734+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:38.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:38.718+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:38.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:38.718+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:38.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:38.718+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:38.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:38.719+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:38.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:38.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:38.562+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:38.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:38.563+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:38.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:38.704+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:38.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:38.704+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:38.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:38.704+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:38.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:38.705+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:40.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:39.613+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26266 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:39.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:39.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:39.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:40.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:39.698+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:40.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:39.606+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:40.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:39.606+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:40.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:39.606+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:40.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:39.607+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:40.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:39.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:40.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:39.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:40.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:39.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:40.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:39.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:40.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:40.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:40.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:41.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:40.654+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:40.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:40.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:40.711+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:41.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:40.711+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:41.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:40.653+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:41.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:40.653+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:41.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:40.653+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:41.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:40.654+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:41.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:41.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:41.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:42.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:41.679+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:42.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:41.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:41.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:41.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:42.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:41.672+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:41.715+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:41.715+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:41.715+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:42.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:41.716+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:42.676+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:42.676+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:42.676+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:43.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:42.677+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:42.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:42.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:42.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:43.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:42.709+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:42.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:42.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:42.718+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:43.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:42.718+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:43.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:43.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:43.710+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:44.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:43.711+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:43.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:43.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:43.739+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:44.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:43.740+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:44.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:43.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:44.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:43.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:44.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:43.719+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:44.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:43.720+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:44.961 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:44.617+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26318 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:44.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:44.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:44.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:44.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:44.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:44.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:44.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:44.687+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:44.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:44.769+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:44.963 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:44.769+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:44.963 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:44.769+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:44.963 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:44.769+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:44.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:44.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:44.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:44.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:44.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:44.725+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:44.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:44.725+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:46.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:45.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:45.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:45.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:46.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:45.698+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:46.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:45.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:45.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:45.767+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:45.768+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:45.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:45.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:45.813+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:46.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:45.814+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:46.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:46.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:46.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:47.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:46.656+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:46.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:46.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:46.863+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:47.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:46.865+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:46.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:46.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:46.792+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:47.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:46.793+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:48.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:47.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:47.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:47.606+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:48.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:47.607+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:47.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:47.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:48.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:47.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:48.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:47.792+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:47.846+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:47.846+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:47.846+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:48.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:47.846+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:48.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:48.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:48.848+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:48.849+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:48.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:48.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:48.762+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:48.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:48.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:48.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:48.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:48.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:48.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:48.625+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:48.977 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:48.627+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:49.882 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:49.619+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26371 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:49.883 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:49.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:49.883 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:49.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:49.884 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:49.811+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:49.884 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:49.812+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:49.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:49.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:49.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:49.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:49.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:49.631+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:49.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:49.632+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:50.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:49.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:49.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:49.880+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:50.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:49.880+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:50.892 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:50.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:50.893 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:50.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:50.893 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:50.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:50.893 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:50.820+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:50.893 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:50.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:50.893 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:50.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:50.894 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:50.611+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:50.894 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:50.612+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:51.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:50.890+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:51.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:50.890+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:51.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:50.890+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:51.300 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:50.891+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:51.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:51.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:51.652+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:52.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:51.652+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:51.857+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:51.857+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:51.857+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:51.858+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:51.854+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:51.854+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:52.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:51.854+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:52.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:51.855+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:52.871 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:52.821+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:52.871 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:52.821+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:52.871 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:52.822+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:52.871 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:52.822+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:52.872 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:52.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:52.872 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:52.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:52.872 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:52.673+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:52.872 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:52.674+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:53.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:52.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:52.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:52.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:52.869+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:54.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:53.855+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:53.855+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:53.855+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:54.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:53.856+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:54.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:53.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:54.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:53.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:54.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:53.654+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:54.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:53.654+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:53.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:53.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:54.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:53.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:54.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:53.838+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:55.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:54.622+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26423 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:55.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:54.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:54.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:54.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:54.832+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:54.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:55.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:54.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:55.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:54.817+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:55.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:54.817+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:55.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:54.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:55.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:54.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:55.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:54.623+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:55.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:54.624+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:56.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:55.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:55.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:55.826+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:56.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:55.827+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:56.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:55.666+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:55.666+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:55.666+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:55.667+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:56.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:55.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:56.065 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:55.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:56.066 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:55.787+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:56.066 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:55.788+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:56.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:56.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:56.859+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:57.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:56.860+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:56.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:56.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:56.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:56.694+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:57.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:56.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:57.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:56.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:57.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:56.782+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:57.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:56.782+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:57.851+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:57.851+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:57.851+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:58.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:57.852+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:57.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:57.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:57.644+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:58.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:57.646+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:57.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:58.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:57.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:58.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:57.818+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:58.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:57.819+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:58.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:58.844+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:58.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:58.844+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:58.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:58.844+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:58.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:58.845+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:58.890 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:58.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:58.890 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:58.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:58.890 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:58.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:58.890 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:58.636+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:58.886+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:59.383 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:58.887+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:59.383 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:58.887+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:08:59.383 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:58.888+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:08:59.911 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:08:59.624+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26476 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:08:59.911 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:59.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:59.912 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:59.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:59.912 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:59.840+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:08:59.912 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:08:59.840+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:08:59.912 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:59.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:59.912 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:59.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:59.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:59.633+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:08:59.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:08:59.634+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:59.909+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:59.909+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:59.909+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:00.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:08:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:08:59.910+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:00.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:00.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:00.883+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:00.884+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:00.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:00.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:00.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:01.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:00.683+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:00.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:00.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:01.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:00.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:01.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:00.888+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:01.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:01.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:01.930+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:01.931+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:01.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:01.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:01.906+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:01.907+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:01.681+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:01.681+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:02.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:01.681+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:02.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:01.682+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:03.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:02.911+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:02.911+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:02.911+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:02.912+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:02.703+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:03.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:02.703+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:03.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:02.703+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:03.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:02.704+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:02.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:02.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:02.930+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:02.931+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:03.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:03.668+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:03.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:03.668+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:03.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:03.668+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:03.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:03.668+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:04.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:03.954+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:03.954+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:03.954+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:04.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:03.954+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:03.920+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:03.920+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:03.920+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:04.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:03.921+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:04.909 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:04.627+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26528 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:04.910 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:04.648+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:04.910 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:04.648+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:04.910 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:04.648+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:04.910 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:04.649+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:05.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:04.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:04.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:04.907+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:05.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:04.908+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:05.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:04.918+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:04.918+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:04.918+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:05.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:04.918+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:05.903 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:05.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:05.903 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:05.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:05.903 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:05.636+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:05.903 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:05.637+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:05.901+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:05.901+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:06.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:05.901+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:06.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:05.902+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:06.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:05.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:06.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:05.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:06.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:05.900+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:06.313 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:05.901+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:06.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:06.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:06.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:06.598+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:06.598+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:06.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:06.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:06.872+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:07.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:06.873+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:07.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:06.871+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:06.871+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:06.871+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:07.312 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:06.871+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:07.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:07.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:07.841+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:07.842+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:07.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:08.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:07.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:08.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:07.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:08.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:07.639+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:08.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:07.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:08.064 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:07.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:08.064 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:07.839+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:08.064 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:07.840+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:08.868 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:08.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:08.868 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:08.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:08.868 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:08.801+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:08.868 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:08.802+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:08.869 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:08.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:08.869 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:08.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:08.869 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:08.604+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:08.869 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:08.605+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:09.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:08.865+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:08.865+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:08.865+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:09.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:08.866+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:09.840 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:09.629+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26577 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:09.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:09.778+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:09.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:09.779+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:09.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:09.779+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:09.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:09.780+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:09.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:09.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:09.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:09.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:09.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:09.562+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:09.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:09.563+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:09.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:09.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:09.837+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:10.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:09.838+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:10.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:10.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:10.544+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:10.544+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:10.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:10.759+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:10.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:10.759+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:10.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:10.759+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:10.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:10.760+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:11.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:10.820+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:10.820+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:10.820+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:11.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:10.820+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:11.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:11.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:11.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:11.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:11.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:11.592+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:11.726 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:11.593+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:11.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:11.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:11.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:11.820+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:11.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:11.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:11.724+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:12.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:11.724+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:12.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:12.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:12.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:13.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:12.606+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:12.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:12.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:12.723+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:12.724+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:12.806+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:13.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:12.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:13.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:12.807+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:13.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:12.807+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:14.042 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:13.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:14.042 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:13.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:14.043 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:13.605+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:14.043 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:13.606+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:14.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:13.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:14.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:13.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:14.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:13.674+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:14.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:13.676+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:14.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:13.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:14.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:13.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:14.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:13.819+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:14.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:13.820+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:15.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:14.632+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26627 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:14.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:14.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:14.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:15.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:14.640+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:15.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:14.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:14.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:14.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:15.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:14.628+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:15.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:14.852+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:15.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:14.852+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:15.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:14.852+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:15.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:14.853+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:15.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:15.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:15.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:16.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:15.587+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:16.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:15.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:16.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:15.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:16.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:15.809+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:16.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:15.810+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:16.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:15.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:16.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:15.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:16.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:15.626+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:16.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:15.627+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:16.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:16.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:16.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:16.619+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:16.619+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:16.579+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:16.579+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:16.579+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:16.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:16.580+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:17.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:16.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:16.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:16.808+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:16.809+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:18.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:17.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:17.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:17.838+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:17.839+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:18.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:17.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:17.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:17.654+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:18.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:17.655+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:17.608+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:17.608+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:18.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:17.608+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:18.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:17.609+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:18.880 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:18.701+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:18.881 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:18.701+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:18.881 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:18.701+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:18.881 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:18.702+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:18.881 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:18.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:18.881 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:18.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:18.882 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:18.593+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:18.882 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:18.594+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:19.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:18.878+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:19.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:18.878+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:19.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:18.878+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:19.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:18.879+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:19.864 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:19.635+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26676 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:19.864 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:19.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:19.864 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:19.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:19.864 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:19.590+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:19.865 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:19.591+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:19.865 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:19.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:19.865 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:19.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:19.865 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:19.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:19.865 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:19.655+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:20.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:19.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:19.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:19.861+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:20.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:19.862+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:20.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:20.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:20.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:20.689+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:20.690+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:20.557+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:20.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:20.557+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:20.557+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:20.557+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:21.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:20.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:20.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:20.879+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:21.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:20.879+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:21.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:21.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:21.733+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:21.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:21.734+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:21.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:21.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:21.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:21.582+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:21.582+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:22.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:21.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:21.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:21.888+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:22.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:21.888+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:22.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:22.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:22.850+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:22.851+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:22.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:22.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:22.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:22.630+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:23.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:22.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:23.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:22.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:23.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:22.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:23.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:22.716+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:24.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:23.817+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:23.817+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:23.817+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:23.818+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:24.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:23.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:24.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:23.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:24.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:23.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:24.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:23.656+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:23.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:23.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:23.678+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:24.063 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:23.679+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:25.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:24.637+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26720 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:24.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:24.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:24.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:24.621+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:24.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:24.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:24.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:24.632+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:24.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:24.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:25.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:24.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:25.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:24.826+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:26.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:25.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:25.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:25.630+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:26.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:25.631+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:25.659+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:25.659+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:25.659+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:26.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:25.660+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:25.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:25.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:25.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:25.854+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:26.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:26.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:26.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:26.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:26.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:26.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:26.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:26.688+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:26.884 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:26.632+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:26.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:26.632+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:26.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:26.632+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:26.885 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:26.633+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:26.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:26.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:26.881+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:27.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:26.882+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:27.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:27.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:27.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:27.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:27.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:27.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:27.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:27.714+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:27.935 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:27.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:27.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:27.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:27.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:27.653+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:27.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:27.655+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:28.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:27.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:27.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:27.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:28.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:27.932+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:28.970 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:28.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:28.970 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:28.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:28.970 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:28.646+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:28.970 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:28.647+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:28.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:28.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:28.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:28.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:28.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:28.743+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:28.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:28.744+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:29.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:28.967+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:28.967+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:28.967+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:29.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:28.968+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:30.013 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:29.640+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26763 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:30.014 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:29.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:30.014 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:29.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:30.014 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:29.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:30.014 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:29.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:30.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:29.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:30.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:29.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:30.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:29.639+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:30.015 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:29.641+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:30.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:30.011+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:30.011+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:30.011+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:30.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:30.012+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:31.012 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:30.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:31.012 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:30.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:31.012 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:30.685+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:31.012 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:30.686+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:31.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:30.731+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:31.013 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:30.731+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:31.013 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:30.731+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:31.013 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:30.731+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:31.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.009+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.009+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.009+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:31.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.010+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:31.993 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:31.683+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:31.994 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:31.683+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:31.994 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:31.683+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:31.994 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:31.683+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:31.994 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:31.672+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:31.994 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:31.672+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:31.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:31.672+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:31.995 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:31.672+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:32.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.991+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:32.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:31.992+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:32.961 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:32.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:32.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:32.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:32.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:32.726+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:32.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:32.727+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:32.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:32.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:32.962 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:32.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:32.963 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:32.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:32.963 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:32.685+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:33.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:32.959+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:32.959+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:32.959+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:33.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:32.960+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:34.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:33.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:34.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:33.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:34.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:33.699+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:34.009 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:33.700+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:34.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:33.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:34.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:33.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:34.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:33.635+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:34.010 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:33.636+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.006+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:34.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.006+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.006+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:34.312 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.007+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:34.984 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:34.643+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26810 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:34.984 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:34.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:34.985 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:34.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:34.985 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:34.620+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:34.985 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:34.621+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:34.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:34.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:34.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:34.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:34.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:34.705+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:34.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:34.706+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:35.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.982+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.982+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.982+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:35.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:34.983+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:35.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:35.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:35.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:35.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:35.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:35.672+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:35.679+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:35.679+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:35.679+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:35.680+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:36.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:35.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:35.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:35.980+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:36.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:35.981+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:36.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:36.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:36.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:36.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:36.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:36.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:36.935 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:36.669+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:36.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:36.660+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:36.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:36.660+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:36.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:36.660+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:36.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:36.661+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:36.936 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:36.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:36.936 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:36.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:36.936 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:36.932+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:37.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:36.933+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:37.978 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:37.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:37.978 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:37.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:37.978 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:37.633+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:37.978 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:37.633+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:37.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:37.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:37.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:37.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:37.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:37.689+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:37.979 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:37.689+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:38.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:37.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:37.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:37.975+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:38.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:37.976+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:38.969 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:38.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:38.970 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:38.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:38.970 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:38.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:38.970 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:38.683+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:38.970 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:38.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:38.970 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:38.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:38.970 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:38.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:38.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:38.640+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:39.038 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-03-12T04:09:39.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:39.044 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-12T04:09:39.045 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-12T04:09:39.057 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T04:09:39.068 INFO:tasks.cephadm:Teardown begin 2024-03-12T04:09:39.068 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-12T04:09:39.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T04:09:39.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:09:39.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:09:39.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:38.967+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:38.967+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:38.967+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:39.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:38.968+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:39.948 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:39.645+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26856 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:39.949 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:39.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:39.949 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:39.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:39.949 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:39.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:39.949 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:39.612+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:39.949 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:39.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:39.949 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:39.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:39.950 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:39.656+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:39.950 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:39.657+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:40.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:39.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:39.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:39.946+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:40.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:39.947+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:40.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:40.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:40.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:40.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:40.913 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:40.687+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:40.914 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:40.688+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:40.914 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:40.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:40.914 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:40.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:40.914 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:40.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:40.914 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:40.632+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:40.915 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:40.910+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:40.915 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:40.910+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:40.915 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:40.910+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:41.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:40.912+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:41.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:41.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:41.955 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:41.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:41.955 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:41.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:41.955 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:41.618+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:41.955 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:41.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:41.955 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:41.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:41.956 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:41.638+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:41.956 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:41.639+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:42.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:09:42.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:41.952+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:41.952+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:41.952+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:42.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:41.953+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:42.926 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:42.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:42.927 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:42.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:42.927 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:42.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:42.927 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:42.628+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:42.927 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:42.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:42.927 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:42.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:42.927 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:42.671+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:42.928 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:42.671+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:42.924+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:42.924+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:42.924+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:43.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:42.925+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:43.973 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:43.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:43.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:43.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:43.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:43.589+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:43.974 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:43.590+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:43.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:43.705+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:43.974 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:43.705+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:43.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:43.705+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:43.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:43.706+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:44.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:43.970+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:43.970+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:43.970+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:44.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:43.971+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:44.943 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:44.647+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26903 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:44.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:44.620+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:44.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:44.620+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:44.943 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:44.620+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:44.944 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:44.621+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:44.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:44.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:44.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:44.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:44.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:44.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:44.944 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:44.683+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:45.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:44.940+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:44.940+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:44.940+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:45.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:44.941+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:45.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:45.696+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:45.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:45.696+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:45.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:45.696+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:45.952 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:45.696+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:45.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:45.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:45.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:45.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:45.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:45.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:45.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:45.608+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:46.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:45.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:45.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:45.949+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:46.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:45.950+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:46.936 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:46.674+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:46.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:46.674+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:46.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:46.674+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:46.937 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:46.675+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:46.937 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:46.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:46.938 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:46.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:46.938 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:46.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:46.938 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:46.595+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:46.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:46.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:46.934+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:47.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:46.935+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:47.918 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:47.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:47.919 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:47.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:47.919 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:47.662+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:47.919 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:47.663+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:47.919 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:47.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:47.919 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:47.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:47.919 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:47.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:47.920 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:47.584+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:48.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:47.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:47.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:47.916+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:48.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:47.917+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:48.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:48.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:48.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:48.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:48.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:48.692+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:48.975 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:48.693+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:48.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:48.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:48.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:48.631+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:48.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:48.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:48.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:48.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:48.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:48.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:48.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:48.876+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:50.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:49.650+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 26955 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:49.694+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:49.694+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:49.694+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:50.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:49.695+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:49.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:49.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:49.663+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:50.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:49.664+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:49.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:50.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:49.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:50.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:49.875+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:50.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:49.875+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:50.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:50.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:50.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:50.651+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:50.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:50.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:50.853+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:50.854+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:51.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:50.713+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:51.065 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:50.713+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:51.065 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:50.713+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:51.066 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:50.713+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:51.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:09:51.176 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:09:51.897 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:51.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:51.897 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:51.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:51.897 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:51.612+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:51.898 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:51.613+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:51.898 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:51.754+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:51.898 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:51.754+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:51.898 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:51.754+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:51.898 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:51.755+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:52.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:51.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:51.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:51.894+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:52.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:51.895+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:52.896 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:52.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:52.896 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:52.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:52.896 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:52.792+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:52.896 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:52.793+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:52.897 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:52.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:52.897 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:52.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:52.897 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:52.614+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:52.897 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:52.614+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:53.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:52.893+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:52.893+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:52.893+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:53.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:52.894+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:53.921 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:53.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:53.921 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:53.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:53.921 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:53.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:53.921 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:53.781+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:53.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:53.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:53.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:53.922 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:53.663+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:54.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:09:54.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:53.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:53.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:53.919+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:54.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:53.919+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:55.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:54.652+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27008 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:54.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:54.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:55.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:54.667+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:55.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:54.669+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:54.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:54.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:54.876+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:55.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:54.877+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:55.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:54.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:55.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:54.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:55.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:54.746+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:55.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:54.747+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:55.679+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:55.679+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:55.679+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:56.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:55.679+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:55.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:55.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:55.758+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:56.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:55.759+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:56.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:55.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:56.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:55.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:56.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:55.869+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:56.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:55.870+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:56.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:56.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:56.634+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:57.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:56.634+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:56.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:56.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:56.738+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:57.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:56.739+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:57.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:56.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:57.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:56.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:57.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:56.868+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:57.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:56.869+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:57.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:57.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:57.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:58.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:57.611+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:57.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:57.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:57.825+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:58.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:57.827+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:58.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:57.691+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:58.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:57.691+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:58.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:57.691+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:58.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:57.692+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:58.833 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:58.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:58.834 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:58.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:58.834 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:58.651+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:58.834 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:58.651+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:58.835 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:58.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:58.835 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:58.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:58.835 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:58.568+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:58.836 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:58.569+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:58.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:58.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:58.831+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:09:59.311 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:58.832+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:09:59.801 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:09:59.655+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27060 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:09:59.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:59.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:59.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:59.522+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:59.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:59.523+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:09:59.801 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:09:59.523+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:09:59.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:59.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:59.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:59.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:59.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:59.682+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:09:59.802 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:09:59.683+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:00.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:59.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:59.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:59.798+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:00.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:09:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:09:59.799+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:00.799 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:00.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:00.799 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:00.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:00.799 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:00.489+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:00.800 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:00.490+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:00.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:00.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:00.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:00.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:00.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:00.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:00.800 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:00.699+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:01.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:00.796+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:00.796+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:00.796+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:01.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:00.797+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:01.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:01.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:01.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:01.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:01.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:01.683+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:01.786 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:01.684+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:01.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:01.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:01.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:01.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:01.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:01.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:01.787 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:01.498+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:02.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:01.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:01.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:01.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:01.784+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:02.528+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:02.528+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:02.528+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:02.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:02.529+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:02.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:02.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:02.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:02.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:02.748+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:02.713+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:02.714+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:02.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:02.714+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:02.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:02.715+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:03.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:03.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:03.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:03.497+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:03.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:03.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:03.714+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:03.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:03.715+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:03.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:03.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:03.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:03.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:03.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:03.699+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:03.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:03.700+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:04.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:04.658+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27077 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:04.743+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:04.743+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:04.743+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:04.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:04.744+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:04.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:04.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:04.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:04.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:04.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:04.535+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:04.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:04.536+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:04.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:04.725+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:04.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:04.725+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:04.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:04.725+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:04.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:04.726+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:05.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:05.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:05.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:05.569+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:05.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:05.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:05.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:05.711+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:05.744+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:05.744+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:06.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:05.744+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:06.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:05.745+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:06.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:06.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:06.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:06.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:06.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:06.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:06.555+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:06.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:06.713+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:06.713+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:06.713+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:06.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:06.714+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:06.682+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:06.682+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:06.682+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:06.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:06.683+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:07.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:07.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:07.538+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:07.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:07.539+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:07.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:07.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:07.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:07.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:07.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:07.652+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:07.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:07.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:07.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:07.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:07.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:07.675+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:07.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:07.676+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:08.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:08.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:08.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:08.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:08.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:08.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:08.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:08.547+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:08.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:08.688+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:08.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:08.688+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:08.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:08.688+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:08.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:08.689+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:08.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:08.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:08.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:08.678+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:08.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:08.679+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:09.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:10:10.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:09.660+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27084 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:09.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:09.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:09.715+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:10.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:09.716+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:09.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:09.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:09.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:10.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:09.576+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:10.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:09.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:10.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:09.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:10.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:09.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:10.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:09.700+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:10.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:10.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:10.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:10.722+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:10.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:10.723+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:10.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:10.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:10.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:10.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:10.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:10.687+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:10.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:10.552+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:10.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:10.552+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:10.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:10.552+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:10.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:10.552+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:11.985 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:11.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:11.986 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:11.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:11.986 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:11.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:11.986 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:11.580+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:11.986 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:11.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:11.986 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:11.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:11.987 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:11.755+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:11.987 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:11.755+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:11.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:11.639+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:11.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:11.639+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:11.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:11.639+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:11.987 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:11.640+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:12.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:12.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:12.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:10:12.386 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:10:12.388 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T04:10:12.388 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T04:10:12.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:12.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:12.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:12.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:12.747+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:12.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:12.748+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:12.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:12.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:12.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:12.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:12.631+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:12.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:12.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:12.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:12.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:12.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:12.547+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:12.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:12.548+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:13.810 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:13.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:13.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:13.745+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:13.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:13.746+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:13.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:13.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:13.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:13.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:13.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:13.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:13.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:13.538+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:13.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:13.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:13.665+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:13.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:13.666+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:14.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:14.663+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27092 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:14.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:14.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:14.513+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:14.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:14.514+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:14.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:14.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:14.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:14.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:14.700+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:14.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:14.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:14.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:14.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:14.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:14.760+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:14.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:14.761+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:15.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:10:15.739 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:15.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:15.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:15.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:15.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:15.476+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:15.740 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:15.477+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:15.740 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:15.697+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:15.740 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:15.697+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:15.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:15.697+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:15.741 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:15.697+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:15.737+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:15.737+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:15.737+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:16.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:15.737+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:16.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:16.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:16.497+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:16.497+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:16.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:16.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:16.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:16.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:16.713+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:16.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:16.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:16.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:16.697+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:16.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:16.698+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:17.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:17.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:17.529+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:17.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:17.530+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:17.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:17.674+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:17.674+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:17.674+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:17.675+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:17.700+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:17.700+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:17.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:17.700+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:17.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:17.701+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:18.524+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:18.524+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:18.524+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:18.525+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:18.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:18.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:18.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:18.631+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:18.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:18.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:18.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:18.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:18.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:18.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:18.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:18.685+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:19.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:19.666+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27099 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:19.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:19.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:19.510+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:19.511+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:19.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:19.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:19.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:19.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:19.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:19.664+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:19.664+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:19.664+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:19.666+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:20.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:20.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:20.534+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:20.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:20.534+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:20.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:20.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:20.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:20.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:20.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:20.582+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:20.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:20.583+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:20.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:20.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:20.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:20.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:20.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:20.686+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:20.815 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:20.687+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:21.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:21.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:21.645+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:22.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:21.645+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:21.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:21.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:21.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:22.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:21.581+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:22.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:21.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:22.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:21.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:22.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:21.562+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:22.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:21.563+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:22.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:22.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:22.596+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:23.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:22.597+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:22.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:22.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:22.693+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:23.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:22.694+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:23.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:22.612+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:23.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:22.612+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:23.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:22.612+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:23.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:22.613+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:23.601+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:23.601+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:24.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:23.601+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:24.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:23.602+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:23.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:24.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:23.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:24.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:23.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:24.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:23.597+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:24.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:23.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:24.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:23.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:24.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:23.684+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:24.064 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:23.684+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:24.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:24.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:25.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:24.668+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27107 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:24.734+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:24.734+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:24.734+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:25.061 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:24.735+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:24.573+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:24.573+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:24.573+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:25.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:24.574+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:24.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:25.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:24.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:25.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:24.584+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:25.063 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:24.585+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:26.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:25.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:25.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:25.592+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:26.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:25.593+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:25.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:25.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:25.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:26.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:25.617+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:26.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:25.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:26.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:25.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:26.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:25.774+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:26.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:25.775+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:26.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:26.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:26.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:27.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:26.599+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:26.645+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:26.645+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:26.645+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:26.646+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:27.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:26.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:27.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:26.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:27.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:26.780+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:27.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:26.781+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:27.522 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:10:28.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:27.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:27.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:27.600+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:28.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:27.601+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:28.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:27.604+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:27.604+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:27.604+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:28.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:27.605+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:28.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:27.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:28.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:27.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:28.062 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:27.799+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:28.063 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:27.800+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:28.845 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:28.609+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:28.845 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:28.609+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:28.845 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:28.609+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:28.845 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:28.610+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:28.845 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:28.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:28.846 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:28.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:28.846 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:28.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:28.846 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:28.559+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:29.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:28.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:28.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:28.842+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:29.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:28.843+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:29.863 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:29.671+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27114 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:29.863 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:29.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:29.864 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:29.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:29.864 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:29.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:29.864 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:29.653+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:29.864 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:29.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:29.864 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:29.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:29.864 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:29.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:29.865 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:29.561+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:30.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:29.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:29.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:29.861+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:30.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:29.862+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:30.888 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:30.643+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:30.888 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:30.643+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:30.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:30.643+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:30.889 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:30.644+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:30.889 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:30.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:30.889 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:30.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:30.889 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:30.573+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:30.889 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:30.574+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:30.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:30.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:30.886+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:31.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:30.886+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:31.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:31.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:31.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:31.556+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:31.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:31.557+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:31.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:31.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:31.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:31.629+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:31.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:31.630+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:31.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:31.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:31.871+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:31.872+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:32.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:32.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:32.885 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:32.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:32.885 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:32.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:32.885 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:32.642+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:32.885 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:32.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:32.885 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:32.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:32.885 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:32.580+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:32.886 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:32.581+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:33.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:32.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:32.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:32.882+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:33.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:32.883+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:33.883 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:33.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:33.883 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:33.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:33.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:33.635+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:33.884 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:33.636+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:33.884 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:33.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:33.884 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:33.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:33.884 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:33.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:33.884 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:33.608+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:34.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:33.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:33.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:33.881+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:34.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:33.881+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:34.905 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:34.673+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27122 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:34.906 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:34.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:34.906 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:34.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:34.906 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:34.591+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:34.906 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:34.592+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:34.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:34.609+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:34.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:34.609+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:34.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:34.609+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:34.907 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:34.610+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:35.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:34.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:34.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:34.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:35.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:34.904+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:35.894 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:35.625+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:35.895 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:35.625+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:35.895 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:35.625+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:35.895 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:35.625+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:35.895 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:35.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:35.895 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:35.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:35.896 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:35.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:35.896 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:35.647+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:36.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:35.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:35.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:35.892+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:36.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:35.893+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:36.898 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:36.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:36.898 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:36.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:36.899 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:36.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:36.899 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:36.636+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:36.899 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:36.644+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:36.899 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:36.644+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:36.899 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:36.644+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:36.899 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:36.645+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:37.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:36.896+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:37.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:36.896+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:37.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:36.896+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:37.185 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:36.897+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:37.905 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:37.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:37.905 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:37.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:37.906 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:37.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:37.906 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:37.678+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:37.906 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:37.632+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:37.906 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:37.632+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:37.906 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:37.632+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:37.906 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:37.633+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:38.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:37.903+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:38.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:37.903+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:38.952 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:38.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:38.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:38.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:38.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:38.669+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:38.953 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:38.670+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:38.953 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:38.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:38.954 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:38.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:38.954 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:38.597+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:38.954 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:38.598+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:39.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:38.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:38.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:38.950+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:39.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:38.951+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:39.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:39.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:39.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:39.676+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27129 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:39.696+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:39.696+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:39.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:39.696+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:39.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:39.697+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:39.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:39.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:39.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:39.549+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:39.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:39.550+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:39.973+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:39.973+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:39.973+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:40.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:39.974+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:40.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:40.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:40.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:40.715+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:40.716+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:40.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:40.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:40.516+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:40.516+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:41.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:40.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:40.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:40.938+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:41.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:40.939+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:41.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:41.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:41.714+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:41.715+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:41.552+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:41.552+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:41.552+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:41.553+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:42.236 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:41.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:42.236 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:41.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:42.236 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:41.968+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:42.236 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:41.969+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:42.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:10:42.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:42.758+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:42.758+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:42.758+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:42.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:42.759+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:42.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:42.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:42.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:42.523+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:42.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:42.524+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:43.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:42.965+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:42.965+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:42.965+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:43.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:42.966+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:43.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:43.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:43.730+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:43.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:43.731+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:43.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:43.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:43.528+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:43.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:43.529+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:44.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.012+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:44.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.013+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:44.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:44.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:44.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:44.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:44.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:44.555+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:44.681 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:44.556+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:44.971 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:44.678+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27137 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:44.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:44.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:44.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:44.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:44.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:44.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:44.972 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:44.709+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:44.972 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:44.972 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:44.972 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.969+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:45.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:44.970+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:45.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:45.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:45.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:10:45.662 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:10:45.663 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T04:10:45.664 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862/remote/smithi118/crash 2024-03-12T04:10:45.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/crash -- . 2024-03-12T04:10:45.727 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862/remote/smithi177/crash 2024-03-12T04:10:45.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:45.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:45.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:45.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:45.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:45.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:45.721+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:45.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:45.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:45.536+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:45.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:45.536+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:46.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.006+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:46.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:46.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:46.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:46.734+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:46.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:46.735+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:46.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:46.498+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:46.498+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:46.498+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:46.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:46.499+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:47.280 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:47.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:47.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.982+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:47.281 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:46.983+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:47.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:47.517+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:47.517+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:47.517+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:47.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:47.518+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:48.021 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:47.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:48.021 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:47.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:48.021 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:47.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:48.021 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:47.688+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:48.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:48.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:48.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:48.018+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:48.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:48.019+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:48.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:10:48.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:48.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:48.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:48.540+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:48.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:48.541+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:48.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:48.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:48.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:48.721+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:48.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:48.722+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:49.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:49.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:49.007+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:49.008+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:50.040 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:49.681+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27144 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:50.040 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:49.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:50.040 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:49.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:50.040 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:49.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:50.041 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:49.694+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:50.041 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:49.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:50.041 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:49.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:50.041 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:49.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:50.041 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:49.590+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:50.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:50.037+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:50.037+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:50.037+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:50.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:50.038+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:50.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:50.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:50.558+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:50.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:50.559+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:50.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:50.729+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:50.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:50.730+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:51.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:51.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:51.061+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:51.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:51.061+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:52.023 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:51.583+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:52.023 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:51.583+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:52.023 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:51.583+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:52.023 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:51.583+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:52.023 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:51.740+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:52.024 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:51.740+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:52.024 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:51.740+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:52.024 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:51.741+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:52.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:52.020+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:52.020+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:52.020+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:52.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:52.021+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:53.031 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:52.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:53.032 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:52.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:53.032 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:52.693+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:53.032 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:52.693+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:53.032 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:52.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:53.032 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:52.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:53.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:52.618+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:53.033 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:52.619+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:53.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.028+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:53.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.029+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:53.998 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:53.736+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:53.998 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:53.736+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:53.998 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:53.736+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:53.998 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:53.738+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:53.999 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:53.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:53.999 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:53.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:53.999 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:53.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:53.999 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:53.643+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:54.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.995+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.995+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.995+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:53.996+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:54.975 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:54.683+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27152 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:54.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:54.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:54.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:54.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:54.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:54.688+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:54.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:54.689+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:54.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:54.691+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:54.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:54.691+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:54.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:54.691+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:54.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:54.692+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:55.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:54.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:54.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:54.972+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:54.973+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:55.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:55.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:55.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:55.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:55.697 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:55.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:55.698 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:55.652+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:56.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:55.695+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:56.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:55.695+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:56.012 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:55.695+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:56.013 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:55.695+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:56.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.010+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.011+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:57.001 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:56.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:57.001 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:56.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:57.001 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:56.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:57.001 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:56.666+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:57.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:56.643+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:57.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:56.643+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:57.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:56.643+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:57.002 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:56.644+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:57.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.998+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:56.999+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:57.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:10:57.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:10:58.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:57.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:58.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:57.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:58.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:57.623+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:58.049 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:57.624+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:58.049 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:57.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:58.050 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:57.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:58.050 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:57.700+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:58.050 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:57.701+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:58.313 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:58.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:58.314 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:58.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:58.314 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:58.046+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:58.314 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:58.047+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:58.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:58.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:58.662+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:58.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:58.662+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:58.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:58.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:58.589+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:58.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:58.589+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:10:59.264 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:59.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:59.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:59.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:59.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:59.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:10:59.265 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:10:59.065+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:59.811 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:10:59.686+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27159 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:10:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:59.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:59.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:59.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:59.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:59.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:10:59.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:10:59.642+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:10:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:59.588+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:59.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:59.588+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:59.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:59.588+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:10:59.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:10:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:10:59.589+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:00.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:00.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:00.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:00.099+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:00.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:00.100+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:00.866 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:11:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:00.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:00.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:00.684+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:01.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:00.684+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:00.620+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:00.620+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:00.620+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:01.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:00.621+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:01.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:01.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:01.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:01.144+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:01.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:01.144+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:01.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:01.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:01.708+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:02.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:01.709+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:02.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:01.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:01.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:01.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:02.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:01.659+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:02.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:02.189+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:02.189+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:02.189+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:02.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:02.190+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:02.690+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:02.690+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:02.690+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:03.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:02.691+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:02.707+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:02.707+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:02.707+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:03.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:02.707+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:03.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:03.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:03.187+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:03.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:03.188+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:03.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:03.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:03.677+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:04.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:03.678+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:04.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:03.664+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:04.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:03.664+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:04.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:03.664+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:04.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:03.664+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:04.155+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:04.155+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:04.155+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:04.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:04.156+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:05.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:04.688+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27167 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:04.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:04.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:04.651+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:05.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:04.652+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:04.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:04.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:04.676+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:05.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:04.676+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:05.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:05.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:05.192+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:05.194+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:06.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:05.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:05.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:05.631+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:05.632+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:05.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:05.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:05.647+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:06.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:05.649+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:06.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:06.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:06.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:06.214+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:06.215+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:07.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:06.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:06.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:06.656+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:07.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:06.657+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:07.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:06.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:07.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:06.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:07.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:06.603+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:07.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:06.604+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:07.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:07.250+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:07.250+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:07.250+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:07.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:07.251+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:08.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:07.650+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:07.650+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:07.650+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:07.651+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:07.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:07.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:07.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:07.575+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:08.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:08.301+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:08.301+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:08.301+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:08.302+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:08.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:08.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:08.975 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:08.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:08.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:08.690+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:08.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:08.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:08.565+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:08.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:08.566+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:09.689 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:09.347+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:09.689 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:09.347+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:09.689 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:09.347+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:09.690 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:09.348+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:09.690 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:09.551+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:09.690 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:09.551+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:09.690 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:09.551+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:09.690 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:09.551+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:09.952 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:09.691+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27174 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:09.952 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:09.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:09.952 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:09.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:09.953 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:09.686+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:09.953 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:09.687+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:10.667 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:10.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:10.668 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:10.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:10.668 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:10.586+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:10.668 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:10.587+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:10.668 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:10.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:10.668 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:10.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:10.668 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:10.362+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:10.669 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:10.363+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:10.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:10.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:10.699+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:11.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:10.700+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:11.691 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:11.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:11.692 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:11.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:11.692 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:11.412+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:11.692 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:11.413+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:11.692 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:11.583+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:11.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:11.583+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:11.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:11.583+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:11.693 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:11.584+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:12.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:11.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:11.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:11.689+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:11.690+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:12.717 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:12.443+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:12.717 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:12.443+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:12.718 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:12.443+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:12.718 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:12.444+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:12.718 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:12.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:12.718 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:12.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:12.718 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:12.575+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:12.718 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:12.576+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:12.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:11:12.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:11:13.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:12.715+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:12.715+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:12.715+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:13.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:12.715+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:13.697 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:13.616+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:13.697 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:13.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:13.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:13.617+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:13.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:13.617+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:13.698 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:13.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:13.698 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:13.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:13.699 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:13.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:13.699 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:13.398+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:13.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:13.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:13.751+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:13.752+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:14.697 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:14.400+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:14.697 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:14.400+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:14.697 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:14.400+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:14.697 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:14.401+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:14.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:14.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:14.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:14.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:14.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:14.624+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:14.698 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:14.625+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:15.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:14.694+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27182 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:14.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:14.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:14.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:15.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:14.711+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:15.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:15.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:15.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:15.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:15.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:15.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:15.703 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:15.608+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:15.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:15.422+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:15.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:15.422+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:15.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:15.422+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:15.704 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:15.422+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:15.934 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:11:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:15.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:15.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:16.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:15.700+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:16.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:15.701+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:16.590+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:16.590+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:16.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:16.590+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:16.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:16.591+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:16.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:16.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:16.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:16.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:16.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:16.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:16.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:16.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:16.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:16.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:16.469+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:16.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:16.470+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:17.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:17.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:17.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:17.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:17.460+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:17.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:17.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:17.618+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:17.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:17.619+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:17.654+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:17.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:17.654+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:17.654+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:17.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:17.655+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:18.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:18.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:18.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:18.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:18.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:18.640+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:18.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:18.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:18.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:18.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:18.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:18.610+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:18.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:18.611+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:18.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:18.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:18.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:18.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:18.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:18.430+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:18.814 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:18.431+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:18.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:11:18.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:11:19.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:11:19.006 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:11:19.007 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T04:11:19.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-12T04:11:19.049 INFO:tasks.cephadm:Compressing logs... 2024-03-12T04:11:19.049 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T04:11:19.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:11:19.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:11:19.114 INFO:teuthology.orchestra.run.smithi118.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-12T04:11:19.115 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T04:11:19.115 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi118.oysrod.log 2024-03-12T04:11:19.116 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi118.wouiai.log 2024-03-12T04:11:19.116 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi118.oysrod.log: gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T04:11:19.116 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mgr.smithi118.zzzuvl.log 2024-03-12T04:11:19.118 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi118.wouiai.log: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-client.ceph-exporter.smithi118.log 2024-03-12T04:11:19.118 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mgr.smithi118.zzzuvl.log: gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mon.smithi118.log 2024-03-12T04:11:19.118 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T04:11:19.119 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-client.ceph-exporter.smithi118.log: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mon.smithi118.log: 90.7% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T04:11:19.120 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T04:11:19.121 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.1.log: 94.5% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-client.ceph-exporter.smithi118.log.gz 2024-03-12T04:11:19.122 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T04:11:19.126 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.0.log: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T04:11:19.135 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.log 2024-03-12T04:11:19.136 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T04:11:19.136 INFO:teuthology.orchestra.run.smithi118.stderr: 83.8% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T04:11:19.137 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.log: 86.9% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T04:11:19.138 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T04:11:19.170 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.2.log: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T04:11:19.337 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mgr.smithi118.zzzuvl.log: file size changed while zipping 2024-03-12T04:11:19.338 INFO:teuthology.orchestra.run.smithi118.stderr: 91.0% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mgr.smithi118.zzzuvl.log.gz 2024-03-12T04:11:19.810 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:19.696+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27189 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:19.656+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:19.656+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:19.656+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:19.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:19.658+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:19.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:19.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:19.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:19.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:19.650+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:19.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:19.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:19.454+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:19.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:19.456+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:20.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:20.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:20.641+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:20.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:20.641+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:20.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:20.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:20.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:20.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:20.672+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:20.472+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:20.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:20.472+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:20.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:20.472+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:20.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:20.473+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:21.339 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mon.smithi118.log: file size changed while zipping 2024-03-12T04:11:21.340 INFO:teuthology.orchestra.run.smithi118.stderr: 96.1% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mon.smithi118.log.gz 2024-03-12T04:11:21.586 INFO:teuthology.orchestra.run.smithi118.stderr: 95.3% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi118.wouiai.log.gz 2024-03-12T04:11:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:21.492+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:21.492+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:21.492+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:21.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:21.493+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:21.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:21.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:21.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:21.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:21.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:21.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:21.813 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:21.594+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:21.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:21.709+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:21.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:21.709+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:21.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:21.709+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:21.814 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:21.710+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:22.146 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:11:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:22.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:22.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:22.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:22.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:22.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:22.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:22.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:22.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:22.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:22.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:22.599+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:22.456+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:22.456+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:22.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:22.456+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:22.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:22.457+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:23.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:23.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:23.593+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:23.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:23.594+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:23.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:23.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:23.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:23.484+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:23.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:23.485+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:23.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:23.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:23.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:23.684+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:23.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:23.684+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:24.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:24.688+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:24.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:24.688+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:24.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:24.688+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:24.714 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:24.689+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:24.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:24.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:24.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:24.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:24.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:24.579+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:24.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:24.580+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:24.715 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:24.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:24.715 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:24.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:24.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:24.459+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:24.716 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:24.460+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:25.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:24.702+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27197 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:25.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:25.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:25.537+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:25.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:25.538+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:25.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:25.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:25.648+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:25.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:25.649+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:25.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:25.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:25.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:25.444+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:25.813 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:25.445+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:26.587 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:26.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:26.588 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:26.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:26.588 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:26.582+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:26.588 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:26.583+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:26.588 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:26.420+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:26.588 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:26.420+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:26.588 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:26.420+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:26.589 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:26.421+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:26.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:26.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:27.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:26.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:27.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:26 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:26.607+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:27.299 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-03-12T04:11:27.299 INFO:teuthology.orchestra.run.smithi118.stderr: 95.0% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T04:11:27.810 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:27.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:27.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:27.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:27.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:27.631+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:27.811 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:27.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:27.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:27.397+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:27.812 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:27.400+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:27.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:27.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:27.594+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:27.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:27 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:27.595+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:28.624 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:28.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:28.624 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:28.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:28.625 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:28.360+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:28.625 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:28.361+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:28.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:28.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:28.621+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:28.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:28.622+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:28.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:28.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:28.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:28.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:28.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:28.655+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:28.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:28 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:28.656+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:29.627 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:29.578+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:29.627 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:29.578+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:29.627 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:29.578+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:29.628 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:29.579+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:29.628 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:29.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:29.628 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:29.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:29.628 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:29.331+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:29.629 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:29.332+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:29.915 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-03-12T04:11:29.915 INFO:teuthology.orchestra.run.smithi118.stderr: 96.5% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T04:11:30.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:29.709+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27204 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:29.625+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:29.625+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:29.625+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:30.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:29 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:29.625+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-03-12T04:11:30.111 INFO:teuthology.orchestra.run.smithi118.stderr: 96.6% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T04:11:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:30.287+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:30.287+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:30.287+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:30.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:30.288+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:30.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:30.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:30.574+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:31.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:30.575+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:31.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:30.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:30.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:30.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:31.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:30 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:30.622+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:31.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:11:31.148 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:11:31.609 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:31.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:31.610 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:31.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:31.610 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:31.321+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:31.610 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:31.321+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:31.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:31.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:31.611+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:32.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:31.612+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:31.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:31.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:31.607+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:32.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:31 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:31.608+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:32.605 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:32.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:32.605 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:32.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:32.605 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:32.335+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:32.605 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:32.336+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:33.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:32.602+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:33.123 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:32.602+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:33.123 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:32.602+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:33.123 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:32.603+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:33.123 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:32.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:33.124 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:32.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:33.124 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:32.630+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:33.124 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:32 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:32.631+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:33.600 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:33.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:33.600 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:33.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:33.600 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:33.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:33.600 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:33.313+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:33.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:33.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:33.598+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:34.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:33.599+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:34.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:33.598+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:34.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:33.598+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:34.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:33.598+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:34.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:33 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:33.600+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:34.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:11:34.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:34.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:34.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:34.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:34.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:34.275+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:34.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:34.277+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:35.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:34.711+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27212 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:34.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:34.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:34.566+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:35.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:34.567+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:34.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:34.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:34.640+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:35.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:34 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:34.641+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:35.545 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:35.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:35.545 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:35.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:35.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:35.274+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:35.546 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:35.275+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:35.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:35.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:35.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:35.545+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:35.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:35.546+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:35.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:35.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:35.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:35.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:35.670+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:35.813 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:35 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:35.671+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:36.687 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:36.319+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:36.688 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:36.319+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:36.688 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:36.319+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:36.688 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:36.320+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:36.688 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:36.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:36.688 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:36.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:36.689 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:36.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:36.689 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:36.568+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:37.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:36.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:36.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:36.685+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:37.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:36 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:36.686+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:37.288+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:37.288+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:37.288+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:37.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:37.288+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:38.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:37.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:37.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:37.658+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:38.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:37.659+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:37.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:38.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:37.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:38.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:37.567+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:38.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:37 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:37.568+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:38.566 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:38.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:38.567 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:38.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:38.567 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:38.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:38.567 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:38.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:38.567 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:38.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:38.567 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:38.313+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:38.568 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:38.314+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:38.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:38.565+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:38.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:38.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:38.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:38.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:38.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:38.637+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:38.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:38 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:38.638+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:39.567 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:39.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:39.567 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:39.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:39.567 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:39.312+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:39.567 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:39.313+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:40.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:39.715+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27219 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:39.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:39.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:39.564+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:40.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:39.565+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:39.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:39.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:39.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:40.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:39 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:39.650+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:40.528 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:40.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:40.529 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:40.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:40.529 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:40.267+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:40.529 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:40.268+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:40.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:40.526+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:40.526+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:40.526+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:40.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:40.527+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:40.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:40.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:40.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:40.642+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:40.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:40 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:40.643+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:41.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:41.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:41.556 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:41.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:41.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:41.289+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:41.557 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:41.290+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:41.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:41.553+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:41.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:41.554+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:41.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:41.555+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:41.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:41.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:41.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:41.615+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:41.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:41 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:41.616+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:42.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:42.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:42.244+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:42.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:42.245+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:42.604+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:42.604+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:42.604+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:43.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:42.605+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:42.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:42.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:42.649+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:43.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:42 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:42.650+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:43.319 INFO:teuthology.orchestra.run.smithi118.stderr: 91.7% -- replaced with /var/log/ceph/9d8e7844-e01e-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi118.oysrod.log.gz 2024-03-12T04:11:43.321 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-12T04:11:43.321 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m24.219s 2024-03-12T04:11:43.322 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m57.494s 2024-03-12T04:11:43.322 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m1.363s 2024-03-12T04:11:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:43.248+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:43.248+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:43.248+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:43.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:43.249+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:43.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:43.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:43.580+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:44.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:43.581+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:44.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:43.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:44.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:43.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:44.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:43.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:44.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:43 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:43.623+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:44.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:44.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:44.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:44.239+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:44.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:44.240+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:45.061 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:44.717+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27227 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:44.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:44.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:45.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:44.622+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:45.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:44.623+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:45.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:44.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:45.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:44.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:45.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:44.569+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:45.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:44 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:44.570+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:45.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:45.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:45.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:45.224+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:45.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:45.224+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:45.616+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:45.616+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:45.616+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:46.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:45.617+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:45.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:45.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:45.671+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:46.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:45 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:45.672+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:46.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:11:46.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:11:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:46.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:46.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:46.213+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:46.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:46.214+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:47.060 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:46.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:46.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:46.652+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:47.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:46.653+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:47.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:46.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:46.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:46.713+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:47.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:46 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:46.714+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:47.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:47.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:47.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:47.174+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:47.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:47.175+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:47.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:47.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:47.687+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:48.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:47.688+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:48.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:47.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:48.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:47.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:48.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:47.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:48.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:47 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:47.734+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:48.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:48.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:48.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:48.136+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:48.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:48.137+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:48.975 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:48.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:48.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:48.716+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:48.976 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:48.717+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:48.779+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:48.779+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:48.976 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:48.779+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:48.977 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:48 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:48.780+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:49.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:11:49.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:49.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:49.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:49.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:49.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:49.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:50.060 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:49.720+0000 7f8eee2cf700 -1 mon.smithi118@0(probing) e2 get_health_metrics reporting 27234 slow ops, oldest is mgrbeacon mgr.smithi118.zzzuvl(9d8e7844-e01e-11ee-95c8-87774f69a715,14608, , 0) 2024-03-12T04:11:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:49.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:49.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:49.673+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:50.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:49.674+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:49.742+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:49.742+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:49.742+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:50.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:49 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:49.743+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:50.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:50.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:50.076+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:50.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:50.077+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:50.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:50.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:50.636+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:51.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:50.637+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:51.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:50.732+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:50.732+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:50.732+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:51.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:50 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:50.733+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:51.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:51.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:51.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:51.084+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:51.563 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:51.085+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:51.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:51.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:51.686+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:52.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:51.687+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:52.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:51.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:51.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:51.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:52.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:51 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:51.734+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:52.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:11:52.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:11:52.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:11:52.351 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:11:52.351 INFO:tasks.cephadm:Removing cluster... 2024-03-12T04:11:52.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 --force 2024-03-12T04:11:52.378 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:52.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:52.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:52.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:52.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:52.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:52.379 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:52.092+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:52.521 INFO:teuthology.orchestra.run.smithi118.stdout:Deleting cluster with fsid: 9d8e7844-e01e-11ee-95c8-87774f69a715 2024-03-12T04:11:52.647 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:52 smithi118 systemd[1]: Stopping Ceph mon.smithi118 for 9d8e7844-e01e-11ee-95c8-87774f69a715... 2024-03-12T04:11:52.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:52.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:52.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:52.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:52.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:52.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:52.923 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:52.645+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:52.923 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:52.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:52.924 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:52.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:52.924 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:52.763+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:52.924 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:52.763+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:53.277 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:52.920+0000 7f8ef12d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi118 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:11:53.278 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:52 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118[29156]: 2024-03-12T04:11:52.920+0000 7f8ef12d5700 -1 mon.smithi118@0(probing) e2 *** Got Signal Terminated *** 2024-03-12T04:11:53.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:53.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:53.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:53.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:53.278 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:53.053+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:53.279 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:53.054+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:53.560 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:53 smithi118 podman[81635]: 2024-03-12 04:11:53.276270007 +0000 UTC m=+0.490311505 container died a26ddd75d886f21bf923031e4897c2faa28d79e845d24b17902134c8170bea36 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=reef-62a11b3, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-12T04:11:53.957 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:53 smithi118 podman[81635]: 2024-03-12 04:11:53.671271911 +0000 UTC m=+0.885313393 container cleanup a26ddd75d886f21bf923031e4897c2faa28d79e845d24b17902134c8170bea36 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=reef-62a11b3) 2024-03-12T04:11:53.957 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:53 smithi118 bash[81635]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118 2024-03-12T04:11:53.958 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:53.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:53.958 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:53.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:53.958 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:53.644+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:53.958 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:53.646+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:53.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:53.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:53.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:53.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:53.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:53.734+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:53.959 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:53 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:53.735+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:54.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:54.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:54.039+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:54.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:54.039+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:54.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:53 smithi118 podman[81649]: 2024-03-12 04:11:53.956159763 +0000 UTC m=+0.675636826 container remove a26ddd75d886f21bf923031e4897c2faa28d79e845d24b17902134c8170bea36 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-mon-smithi118, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-62a11b3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-12T04:11:54.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:54 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118.service: Deactivated successfully. 2024-03-12T04:11:54.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:54 smithi118 systemd[1]: Stopped Ceph mon.smithi118 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T04:11:54.312 INFO:journalctl@ceph.mon.smithi118.smithi118.stdout:Mar 12 04:11:54 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@mon.smithi118.service: Consumed 4min 57.481s CPU time. 2024-03-12T04:11:54.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:54.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:54.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:54.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:54.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:54.639+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:54.954 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:54.639+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:54.955 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:54.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:54.955 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:54.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:54.955 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:54.691+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:54.955 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:54 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:54.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:55.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:55.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:55.006+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:55.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:55.007+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:55.671 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:55.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:55.672 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:55.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:55.672 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:55.668+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:56.010 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:55.669+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:56.010 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:55.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:56.011 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:55.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:56.011 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:55.681+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:56.011 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:55 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:55.682+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:56.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:56.052+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:56.052+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:56.052+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:56.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:56.052+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:56.916 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:56.653+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:56.916 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:56.653+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:56.916 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:56.653+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:56.917 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:56.653+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:56.917 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:56.664+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:56.917 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:56.664+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:56.917 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:56.664+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:56.917 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:56 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:56.665+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:57.037+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:57.037+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:57.037+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:57.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:57.038+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:57.631+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:57.631+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:57.631+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:57.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:57.632+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:57.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:57.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:57.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:57.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:57.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:57.642+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:57.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:57 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:57.643+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:58.216 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:58.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:58.216 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:58.025+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:58.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:58.025+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:58.217 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:58.025+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:58.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:58.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:58.616+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:58.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:58.617+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:11:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:58.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:58.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:58.640+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:58.812 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:58 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:58.640+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:59.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:59.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:59.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:59.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:59.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:59.051+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:11:59.146 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:11:59.052+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:59.904 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:59.622+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:59.904 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:59.622+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:59.904 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:59.622+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:11:59.905 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:11:59.623+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:11:59.905 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:59.602+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:59.905 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:59.602+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:59.905 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:59.602+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:11:59.905 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:11:59 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:11:59.603+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:00.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:00.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:00.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:00.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:00.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:00.060+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:00.175 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:00.061+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:00.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:00.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:00.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:00.627+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:00.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:00.627+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:00.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:00.811 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:00.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:00.603+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:00.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:00 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:00.604+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:01.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:01.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:01.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:01.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:01.290 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:01.091+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:01.291 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:01.091+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:01.874 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:01.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:01.875 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:01.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:01.875 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:01.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:01.875 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:01.665+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:01.875 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:01.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:01.875 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:01.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:01.875 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:01.576+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:01.876 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:01 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:01.577+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:02.275 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:02.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:02.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:02.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:02.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:02.111+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:02.276 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:02.112+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:02.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:02.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:02.560+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:03.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:02.561+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:02.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:02.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:02.630+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:03.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:02 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:02.631+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:03.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:03.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:03.514 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:03.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:03.515 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:03.110+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:03.515 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:03.110+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:03.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:03.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:03.815 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:03.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:03.816 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:03.655+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:03.816 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:03.656+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:03.816 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:03.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:03.817 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:03.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:03.817 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:03.605+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:03.817 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:03 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:03.606+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:04.452 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:04.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:04.452 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:04.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:04.452 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:04.150+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:04.452 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:04.151+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:04.452 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 systemd[1]: Stopping Ceph osd.0 for 9d8e7844-e01e-11ee-95c8-87774f69a715... 2024-03-12T04:12:04.453 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:04.449+0000 7fe57dbde700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:12:04.453 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:04.449+0000 7fe57dbde700 -1 osd.0 48 *** Got signal Terminated *** 2024-03-12T04:12:04.453 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:04.449+0000 7fe57dbde700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:12:04.810 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:04.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:04.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:04.664+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:04.811 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:04.664+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:04.624+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:04.624+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:04.624+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:04.812 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:04 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:04.625+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:05.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:05.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:05.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:05.108+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:05.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:05.109+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:06.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:05.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:05.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:05.659+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:06.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:05.660+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:06.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:05.623+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:05.623+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:05.623+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:06.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:05 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:05.624+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:06.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:06.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:06.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:06.064+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:06.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:06.064+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:07.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:06.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:07.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:06.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:07.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:06.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:07.043 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:06.673+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:07.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:06.668+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:07.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:06.668+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:07.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:06.668+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:07.044 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:06 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:06.669+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:07.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:07.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:07.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:07.040+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:07.311 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:07.041+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:08.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:07.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:07.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:07.710+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:08.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:07.711+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:07.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:08.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:07.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:07.672+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:08.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:07 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:07.673+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:08.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:08.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:08.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:08.067+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:08.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:08.068+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:09.026 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:08.706+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:09.027 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:08.706+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:09.027 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:08.706+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:09.027 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:08.707+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:09.027 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:08.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:09.027 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:08.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:09.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:08.737+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:09.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:08 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:08.738+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:09.028 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:09.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:09.028 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:09.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:09.028 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:09.024+0000 7fe57a9d5700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:24.933291+0000 (oldest deadline 2024-03-12T04:07:44.933291+0000) 2024-03-12T04:12:09.310 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0[48018]: 2024-03-12T04:12:09.025+0000 7fe57a9d5700 -1 osd.0 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5480 3.1f 3:fcc08ce2:::1000000389a.00000000:head [write 0~8834 in=8834b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:10.060 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:09 smithi118 podman[82598]: 2024-03-12 04:12:09.904120976 +0000 UTC m=+5.572233980 container died 9436f155422d772473dff0eb5e2fa00e2b6dddfd2e4859a9e5dead808b8eb09c (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-62a11b3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:12:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:09.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:09.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:09.671+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:10.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:09.672+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:09.735+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:09.736+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:09.736+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:10.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:09 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:09.736+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:10.560 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:10 smithi118 podman[82598]: 2024-03-12 04:12:10.301058538 +0000 UTC m=+5.969171546 container cleanup 9436f155422d772473dff0eb5e2fa00e2b6dddfd2e4859a9e5dead808b8eb09c (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=reef-62a11b3, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T04:12:10.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:10 smithi118 bash[82598]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0 2024-03-12T04:12:10.923 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:10 smithi118 podman[82616]: 2024-03-12 04:12:10.560978829 +0000 UTC m=+0.656933089 container remove 9436f155422d772473dff0eb5e2fa00e2b6dddfd2e4859a9e5dead808b8eb09c (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0, RELEASE=reef-62a11b3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-12T04:12:10.924 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:10.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:10.924 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:10.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:10.924 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:10.665+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:10.924 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:10.666+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:10.924 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:10.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:10.924 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:10.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:10.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:10.783+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:10.925 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:10 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:10.785+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:11.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:10 smithi118 podman[82705]: 2024-03-12 04:12:10.825598969 +0000 UTC m=+0.018463706 image pull fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6 2024-03-12T04:12:11.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:10 smithi118 podman[82705]: 2024-03-12 04:12:10.971624273 +0000 UTC m=+0.164489039 container create 810da0b0647a9beccdc52759814f82ec1a7ed3e0bc9c34276c29e4c4efb3d1d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-62a11b3, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-12T04:12:11.225 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:11 smithi118 podman[82705]: 2024-03-12 04:12:11.223293372 +0000 UTC m=+0.416158104 container init 810da0b0647a9beccdc52759814f82ec1a7ed3e0bc9c34276c29e4c4efb3d1d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-62a11b3, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T04:12:11.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:11 smithi118 podman[82705]: 2024-03-12 04:12:11.226824369 +0000 UTC m=+0.419689101 container start 810da0b0647a9beccdc52759814f82ec1a7ed3e0bc9c34276c29e4c4efb3d1d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=reef-62a11b3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T04:12:11.525 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:11 smithi118 podman[82705]: 2024-03-12 04:12:11.331850692 +0000 UTC m=+0.524715442 container attach 810da0b0647a9beccdc52759814f82ec1a7ed3e0bc9c34276c29e4c4efb3d1d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=reef-62a11b3, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:12:11.794 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:11 smithi118 podman[82705]: 2024-03-12 04:12:11.524205071 +0000 UTC m=+0.717069816 container died 810da0b0647a9beccdc52759814f82ec1a7ed3e0bc9c34276c29e4c4efb3d1d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=reef-62a11b3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T04:12:11.794 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:11.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:11.794 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:11.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:11.795 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:11.661+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:11.795 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:11.661+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:12.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:11.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:11.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:11.791+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:12.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:11 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:11.792+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:12.561 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 systemd[1]: Stopping Ceph osd.1 for 9d8e7844-e01e-11ee-95c8-87774f69a715... 2024-03-12T04:12:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:12 smithi118 podman[82731]: 2024-03-12 04:12:12.145037075 +0000 UTC m=+0.612235010 container remove 810da0b0647a9beccdc52759814f82ec1a7ed3e0bc9c34276c29e4c4efb3d1d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=reef-62a11b3) 2024-03-12T04:12:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:12 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-12T04:12:12.561 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:12 smithi118 systemd[1]: Stopped Ceph osd.0 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T04:12:12.562 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 12 04:12:12 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.0.service: Consumed 22min 57.188s CPU time. 2024-03-12T04:12:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:12.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:12.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:12.650+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:12.650+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:12.664+0000 7f3509d47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:12:13.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:12.664+0000 7f3509d47700 -1 osd.1 48 *** Got signal Terminated *** 2024-03-12T04:12:13.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:12.664+0000 7f3509d47700 -1 osd.1 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:12:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:12.789+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:12.789+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:12.789+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:13.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:12 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:12.789+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:14.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:13.785+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:13.785+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:13.785+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:14.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:13.785+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:13.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:14.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:13.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:14.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:13.607+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:14.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:13 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:13.608+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:14.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:14.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:14.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:14.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:14.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:14.649+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:14.962 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:14.650+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:14.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:14.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:14.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:14.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:14.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:14.746+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:14.963 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:14 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:14.747+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:15.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:15.600+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:15.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:15.600+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:15.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:15.600+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:15.971 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:15.600+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:15.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:15.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:15.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:15.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:15.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:15.733+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:15.972 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:15 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:15.733+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:16.626+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:16.626+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:16.626+0000 7f3503337700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:28.164109+0000 (oldest deadline 2024-03-12T04:07:48.164109+0000) 2024-03-12T04:12:17.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1[55467]: 2024-03-12T04:12:16.626+0000 7f3503337700 -1 osd.1 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14536.0:5479 3.0 3:03324f73:::10000003a30.00000000:head [write 0~768 in=768b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-12T04:12:17.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:16.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:16.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:16.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:17.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:16 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:16.725+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:17.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:17.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:17.690+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:18.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:17 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:17.691+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:18.660 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:18 smithi118 podman[82821]: 2024-03-12 04:12:18.334852332 +0000 UTC m=+5.805668809 container died 38fd187fe41b11e53aa8cbb83d44367191248e53b1777ac13f7fa372e6027770 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-62a11b3, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-12T04:12:19.027 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:18 smithi118 podman[82821]: 2024-03-12 04:12:18.75806755 +0000 UTC m=+6.228884023 container cleanup 38fd187fe41b11e53aa8cbb83d44367191248e53b1777ac13f7fa372e6027770 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-62a11b3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-12T04:12:19.028 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:18 smithi118 bash[82821]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1 2024-03-12T04:12:19.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:18.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:19.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:18.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:19.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:18.724+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:19.028 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:18 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:18.724+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:19.310 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:19 smithi118 podman[82841]: 2024-03-12 04:12:19.026229024 +0000 UTC m=+0.690898661 container remove 38fd187fe41b11e53aa8cbb83d44367191248e53b1777ac13f7fa372e6027770 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=reef-62a11b3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-12T04:12:19.714 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:19 smithi118 podman[82932]: 2024-03-12 04:12:19.328279118 +0000 UTC m=+0.018342097 image pull fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6 2024-03-12T04:12:19.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:19 smithi118 podman[82932]: 2024-03-12 04:12:19.460717739 +0000 UTC m=+0.150780699 container create e91056039f017393530eadbf19252ad3b13f2d0ad851b742e8c335cf71763f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-62a11b3, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-12T04:12:19.715 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:19 smithi118 podman[82932]: 2024-03-12 04:12:19.713396625 +0000 UTC m=+0.403459597 container init e91056039f017393530eadbf19252ad3b13f2d0ad851b742e8c335cf71763f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=reef-62a11b3, org.label-schema.vendor=CentOS) 2024-03-12T04:12:20.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:19.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:20.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:19.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:20.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:19.727+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:20.016 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:19 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:19.727+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:20.017 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:19 smithi118 podman[82932]: 2024-03-12 04:12:19.716529276 +0000 UTC m=+0.406592243 container start e91056039f017393530eadbf19252ad3b13f2d0ad851b742e8c335cf71763f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-62a11b3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:12:20.017 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:19 smithi118 podman[82932]: 2024-03-12 04:12:19.821989847 +0000 UTC m=+0.512052816 container attach e91056039f017393530eadbf19252ad3b13f2d0ad851b742e8c335cf71763f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=reef-62a11b3, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-12T04:12:20.311 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:20 smithi118 podman[82932]: 2024-03-12 04:12:20.015037142 +0000 UTC m=+0.705100096 container died e91056039f017393530eadbf19252ad3b13f2d0ad851b742e8c335cf71763f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=reef-62a11b3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-12T04:12:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:20 smithi118 podman[82958]: 2024-03-12 04:12:20.684873553 +0000 UTC m=+0.664207647 container remove e91056039f017393530eadbf19252ad3b13f2d0ad851b742e8c335cf71763f58 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=reef-62a11b3, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:12:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:20 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-12T04:12:21.061 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:20 smithi118 systemd[1]: Stopped Ceph osd.1 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T04:12:21.062 INFO:journalctl@ceph.osd.1.smithi118.stdout:Mar 12 04:12:20 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.1.service: Consumed 23min 35.508s CPU time. 2024-03-12T04:12:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:20.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:20.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:20.756+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:20 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:20.758+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:21.062 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:20 smithi118 systemd[1]: Stopping Ceph osd.2 for 9d8e7844-e01e-11ee-95c8-87774f69a715... 2024-03-12T04:12:21.560 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:21.170+0000 7f018ba25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:12:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:21.170+0000 7f018ba25700 -1 osd.2 48 *** Got signal Terminated *** 2024-03-12T04:12:21.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:21.170+0000 7f018ba25700 -1 osd.2 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:12:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:21.743+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:21.743+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:21.743+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:22.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:21 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:21.744+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:23.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:22.698+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:22.698+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:22.698+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:23.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:22 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:22.699+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:24.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:23.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:23.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:23.728+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:24.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:23 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:23.729+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:25.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:24.754+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:24.754+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:24.754+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:25.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:24 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:24.754+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:26.060 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:25.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6804 osd.3 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:25.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6812 osd.4 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:25.795+0000 7f0186017700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.177:6820 osd.5 ever on either front or back, first ping sent 2024-03-12T04:07:09.613123+0000 (oldest deadline 2024-03-12T04:07:29.613123+0000) 2024-03-12T04:12:26.061 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:25 smithi118 ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2[62870]: 2024-03-12T04:12:25.796+0000 7f0186017700 -1 osd.2 48 get_health_metrics reporting 256 slow ops, oldest is osd_op(mds.0.13:666 2.e 2:715f5c78:::200.00000013:head [write 1586070~1162652 in=1162652b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-03-12T04:12:26.932 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:26 smithi118 podman[83035]: 2024-03-12 04:12:26.666316371 +0000 UTC m=+5.636527576 container died a5856da80958787c66b40642095e893a8a6743676871bd99c3ba05b4a7e4211f (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=reef-62a11b3, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-12T04:12:27.200 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 podman[83035]: 2024-03-12 04:12:27.005194965 +0000 UTC m=+5.975406168 container cleanup a5856da80958787c66b40642095e893a8a6743676871bd99c3ba05b4a7e4211f (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=reef-62a11b3, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-12T04:12:27.200 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 bash[83035]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2 2024-03-12T04:12:27.561 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 podman[83052]: 2024-03-12 04:12:27.198533777 +0000 UTC m=+0.527451265 container remove a5856da80958787c66b40642095e893a8a6743676871bd99c3ba05b4a7e4211f (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=reef-62a11b3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-12T04:12:27.887 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 podman[83143]: 2024-03-12 04:12:27.518050454 +0000 UTC m=+0.017972601 image pull fa4cc2e02803ce1e0678e31bb8bc98e8cb1627995156df45f31641fe2429a464 quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6 2024-03-12T04:12:27.887 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 podman[83143]: 2024-03-12 04:12:27.624432027 +0000 UTC m=+0.124354159 container create b9f2f59ea0ccf30ed184d60958d16122656bea1d9c99ebbd74d54146a008fce5 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-62a11b3, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-12T04:12:27.887 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 podman[83143]: 2024-03-12 04:12:27.810398114 +0000 UTC m=+0.310320259 container init b9f2f59ea0ccf30ed184d60958d16122656bea1d9c99ebbd74d54146a008fce5 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=reef-62a11b3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:12:27.887 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 podman[83143]: 2024-03-12 04:12:27.813652503 +0000 UTC m=+0.313574643 container start b9f2f59ea0ccf30ed184d60958d16122656bea1d9c99ebbd74d54146a008fce5 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=reef-62a11b3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:12:27.887 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:27 smithi118 podman[83143]: 2024-03-12 04:12:27.885153461 +0000 UTC m=+0.385075601 container attach b9f2f59ea0ccf30ed184d60958d16122656bea1d9c99ebbd74d54146a008fce5 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=reef-62a11b3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:12:28.310 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:28 smithi118 podman[83143]: 2024-03-12 04:12:28.100459016 +0000 UTC m=+0.600381140 container died b9f2f59ea0ccf30ed184d60958d16122656bea1d9c99ebbd74d54146a008fce5 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=reef-62a11b3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-12T04:12:29.004 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:28 smithi118 podman[83169]: 2024-03-12 04:12:28.640182707 +0000 UTC m=+0.528846730 container remove b9f2f59ea0ccf30ed184d60958d16122656bea1d9c99ebbd74d54146a008fce5 (image=quay.ceph.io/ceph-ci/ceph@sha256:4fcc703e1f8ba79df609e05789c3603efe338f9b74b627b21043f6e2b3159ae6, name=ceph-9d8e7844-e01e-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=reef-62a11b3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5) 2024-03-12T04:12:29.004 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:28 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-12T04:12:29.004 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:28 smithi118 systemd[1]: Stopped Ceph osd.2 for 9d8e7844-e01e-11ee-95c8-87774f69a715. 2024-03-12T04:12:29.004 INFO:journalctl@ceph.osd.2.smithi118.stdout:Mar 12 04:12:28 smithi118 systemd[1]: ceph-9d8e7844-e01e-11ee-95c8-87774f69a715@osd.2.service: Consumed 23min 43.970s CPU time. 2024-03-12T04:12:30.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:12:30.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:12:33.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:12:42.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:12:42.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:12:45.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:12:57.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:12:57.826 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:00.899 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:13:03.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:13:03.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:03.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:13:03.967 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:13:03.967 INFO:tasks.cephadm:Teardown complete 2024-03-12T04:13:03.967 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 1613, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 1867, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 1870, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 1746, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 200, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephadm.py", line 368, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:13:03.992 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-12T04:13:04.004 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-12T04:13:04.021 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-12T04:13:04.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-12T04:13:04.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:13:04.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:07.106 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:13:16.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:13:16.108 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:19.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:13:31.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:13:31.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:34.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:13:37.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:13:37.245 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:37.314 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:13:37.314 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:13:37.368 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:13:37.389 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-12T04:13:37.400 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-12T04:13:37.401 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:13:37.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:13:37.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:37.423 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:13:37.428 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:13:37.428 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-12T04:13:37.428 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 22 -479us[ -479us] +/- 97ms 2024-03-12T04:13:37.428 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -1525us[-1501us] +/- 54ms 2024-03-12T04:13:37.428 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +2277us[+2277us] +/- 69ms 2024-03-12T04:13:37.428 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:13:40.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:13:49.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:13:49.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:13:52.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:04.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:04.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:07.618 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:10.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:10.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:10.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:10.687 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:14:10.687 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:14:10.702 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-12T04:14:10.713 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-12T04:14:10.714 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-12T04:14:10.728 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T04:14:10.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:10.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:13.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:22.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:22.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:25.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:37.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:37.953 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:41.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:44.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:44.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:44.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:44.094 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:14:44.095 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:14:44.104 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-12T04:14:44.117 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-12T04:14:44.128 INFO:teuthology.task.internal:Duration was 4233.275347 seconds 2024-03-12T04:14:44.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-12T04:14:44.138 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-12T04:14:44.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T04:14:44.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:44.145 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:44.192 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:14:47.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:14:56.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:14:56.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:14:59.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:15:11.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:15:11.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:15:14.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:15:17.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:15:17.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:15:17.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:15:17.407 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:15:17.407 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:15:17.422 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-12T04:15:17.437 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-12T04:15:17.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T04:15:17.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:15:17.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:15:20.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:15:29.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:15:29.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:15:32.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:15:44.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:15:44.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:15:47.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:15:50.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:15:50.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:15:50.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:15:50.750 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:15:50.751 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:15:50.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-12T04:15:50.778 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:15:50.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:15:50.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:15:50.813 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-03-12T04:15:53.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:02.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:02.856 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:05.918 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:17.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:17.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:20.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:23.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:23.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:24.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:24.062 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:16:24.063 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:16:24.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-12T04:16:24.090 INFO:teuthology.task.internal:Transferring archived files... 2024-03-12T04:16:24.091 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862/remote/smithi118 2024-03-12T04:16:24.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T04:16:24.134 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862/remote/smithi177 2024-03-12T04:16:24.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:24.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:27.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:36.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:36.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:39.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:51.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:51.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:54.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:57.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:57.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:57.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:16:57.406 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:16:57.407 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:16:57.424 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-12T04:16:57.436 INFO:teuthology.task.internal:Not uploading archives. 2024-03-12T04:16:57.436 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-12T04:16:57.472 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-12T04:16:57.472 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T04:16:57.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:16:57.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:16:57.495 INFO:teuthology.orchestra.run.smithi118.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 12 04:13 /home/ubuntu/cephtest 2024-03-12T04:16:57.495 INFO:teuthology.orchestra.run.smithi118.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 12 04:15 /home/ubuntu/cephtest/archive 2024-03-12T04:16:57.495 INFO:teuthology.orchestra.run.smithi118.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 03:04 /home/ubuntu/cephtest/archive/syslog 2024-03-12T04:16:57.495 INFO:teuthology.orchestra.run.smithi118.stdout: 265301 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 12 03:04 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T04:16:57.495 INFO:teuthology.orchestra.run.smithi118.stdout: 265300 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 12 03:04 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T04:16:57.495 INFO:teuthology.orchestra.run.smithi118.stdout: 531399 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 04:14 /home/ubuntu/cephtest/archive/audit 2024-03-12T04:16:57.496 INFO:teuthology.orchestra.run.smithi118.stdout: 531402 180 -rw------- 1 ubuntu ubuntu 183901 Mar 12 04:14 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-12T04:16:57.496 INFO:teuthology.orchestra.run.smithi118.stdout: 265386 212 -rwxr-xr-x 1 ubuntu ubuntu 214739 Mar 12 03:14 /home/ubuntu/cephtest/cephadm 2024-03-12T04:16:57.497 INFO:teuthology.orchestra.run.smithi118.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-12T04:17:00.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:17:09.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:17:09.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:17:12.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:17:24.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:17:24.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:17:27.678 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:17:30.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-12T04:17:30.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T04:17:30.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-12T04:17:30.750 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-12T04:17:30.751 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi177 2024-03-12T04:17:30.767 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-12T04:17:30.780 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-12T04:17:30.913 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-03-12T04:17:30.913 INFO:teuthology.nuke:Checking targets against current locks 2024-03-12T04:17:30.946 DEBUG:teuthology.nuke:shortname: smithi118 2024-03-12T04:17:30.946 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T04:17:30.971 DEBUG:teuthology.nuke:shortname: smithi177 2024-03-12T04:17:30.971 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T04:17:30.975 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:54:37.197875', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHVF0YvyzhzCRwLRGv3CVYU883ajunQVhb3K+nZMFHHWog9mEDJNR9E5d5OndRoXiAWhGhwKsKgPsbrq18T5bR4='} 2024-03-12T04:17:30.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591862', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 02:54:37.196400', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGdeBfwI5tI5+DIeBLgsIfbfseyNiNrXbFqN85+hvWLqJoRwTeCkG2KHpZat8/o6wlt81KlndtaoaTWXHx7K/QA='} 2024-03-12T04:17:31.026 INFO:teuthology.orchestra.console.smithi118:Power off 2024-03-12T04:17:31.026 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T04:17:31.051 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2024-03-12T04:17:31.051 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:17:31.065 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-12T04:17:31.066 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T04:17:31.076 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-12T04:17:31.090 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-12T04:17:31.091 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:17:31.116 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-12T04:17:35.078 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:17:35.117 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:17:35.189 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-12T04:17:35.228 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-12T04:17:39.194 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:17:39.229 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:17:39.305 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2024-03-12T04:17:39.341 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-12T04:17:39.407 INFO:teuthology.orchestra.console.smithi118:Power off completed 2024-03-12T04:17:43.344 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:17:43.455 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-12T04:17:43.556 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-12T04:17:43.662 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 4233.2753469944 failure_reason: 'Command failed on smithi118 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c1161e9a464c4e0d82845819cd66a215 status: fail success: false 2024-03-12T04:17:43.663 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T04:17:43.753 INFO:teuthology.run:FAIL